Oct 30 11:29:20.794623 ip-10-0-143-18 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 11:29:21.178634 ip-10-0-143-18 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:21.178634 ip-10-0-143-18 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 11:29:21.178634 ip-10-0-143-18 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:21.178634 ip-10-0-143-18 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:29:21.183065 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.182984    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.183065 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183064    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183069    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183073    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183077    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183080    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183083    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183086    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183089    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183092    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183096    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183101    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183105    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183107    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183110    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183113    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183116    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183119    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183129    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183132    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.183154 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183135    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183137    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183140    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183142    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183145    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183148    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183150    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183153    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183156    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183159    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183162    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183171    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183174    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183177    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183179    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183182    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183185    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183188    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183190    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.183615 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183193    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183196    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183198    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183201    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183204    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183206    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183209    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183211    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183214    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183216    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183220    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183223    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183226    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183230    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183234    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183237    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183240    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183243    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183245    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.184060 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183248    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183251    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183254    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183257    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183259    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183262    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183264    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183267    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183269    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183272    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183275    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183277    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183280    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183283    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183286    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183288    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183291    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183294    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183296    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.184562 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183299    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183302    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183304    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183307    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183310    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183312    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183315    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183318    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183320    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183323    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183326    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183340    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183343    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183346    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183348    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183351    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183353    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183356    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183359    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183363    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.185008 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183365    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183368    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183371    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183373    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183376    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183379    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183381    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183384    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.183386    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184163    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184171    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184173    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184176    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184179    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184182    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184185    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184188    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184190    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184193    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.185507 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184196    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184199    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184202    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184204    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184207    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184210    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184213    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184216    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184218    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184221    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184223    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184226    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184228    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184231    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184233    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184236    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184238    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184240    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184243    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184245    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.185972 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184248    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184250    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184253    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184256    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184259    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184261    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184264    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184267    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184270    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184272    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184275    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184277    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184280    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184283    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184286    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184290    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184295    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184298    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184301    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.186457 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184303    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184306    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184308    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184311    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184314    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184316    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184319    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184323    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184326    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184343    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184346    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184349    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184352    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184354    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184357    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184360    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184363    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184365    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184368    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.186915 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184371    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184373    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184376    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184379    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184381    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184384    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184387    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184390    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184393    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184395    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184398    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184400    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184403    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184406    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184408    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184411    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184414    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184416    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184420    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184422    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.187465 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184425    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184427    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184430    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184432    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184435    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184437    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184440    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184443    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184445    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184447    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184450    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184453    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184457    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184459    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184465    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184468    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184470    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184945    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184950    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.188798 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184953    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184955    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184958    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184961    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184963    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184967    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184970    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184972    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184975    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184978    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184980    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184983    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184985    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184988    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184990    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184993    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184996    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.184999    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185001    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185004    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.189300 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185006    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185009    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185012    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185015    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185018    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185020    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185024    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185026    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185029    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185032    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185035    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185037    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185040    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185042    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185045    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185047    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185050    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185052    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185055    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185057    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.189799 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185060    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185062    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185065    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185067    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185070    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185073    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185075    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185078    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185080    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185083    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185086    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185088    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185091    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185093    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185096    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185099    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185102    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185105    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185107    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185110    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.190293 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185113    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185116    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185121    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185123    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185126    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185130    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185134    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185138    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185141    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185144    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185147    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185151    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185153    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185156    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185159    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185161    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185164    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.190811 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185166    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185170    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185174    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185176    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185179    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185181    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185184    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185187    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185189    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185192    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185195    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185197    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185200    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185203    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185206    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185208    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185211    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185214    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185217    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.191237 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185219    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185222    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185224    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185227    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185229    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185232    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.185234    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186398    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186409    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186417    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186421    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186426    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186429    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186434    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186438    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186441    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186445    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186448    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186452    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186455    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186459    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186462    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186465    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 11:29:21.191724 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186468    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186472    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186477    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186480    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186483    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186486    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186490    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186494    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186497    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186500    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186503    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186506    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186510    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186513    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186516    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186519    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186523    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186526    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186529    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186532    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186535    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186538    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186543    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186546    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 11:29:21.192292 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186552    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186555    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186558    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186562    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186566    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186569    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186572    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186575    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186579    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186582    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186585    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186588    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186594    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186597    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186601    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186604    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186608    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186611    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186614    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186617    2581 flags.go:64] FLAG: --help="false"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186620    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186624    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186627    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 11:29:21.192888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186630    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186634    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186637    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186640    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186644    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186647    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186650    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186653    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186657    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186661    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186664    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186667    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186670    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186673    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186676    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186679    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186682    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186686    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186691    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186694    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186698    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186702    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186705    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186709    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 11:29:21.193515 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186712    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186715    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186719    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186723    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186727    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186730    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186733    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186736    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186739    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186743    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186747    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186750    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186757    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186761    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186764    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186767    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186770    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186774    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186777    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186780    2581 flags.go:64] FLAG: --port="10250"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186783    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186786    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dbe10fdb81214965"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186790    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186793    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186796    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186799    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 11:29:21.194096 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186803    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186806    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186809    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186814    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186818    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186822    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186825    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186828    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186831    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186834    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186837    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186840    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186843    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186846    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186850    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186853    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186856    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186859    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186862    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186865    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186868    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186872    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186875    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186879    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186884    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 11:29:21.194718 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186887    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186890    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186894    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186897    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186900    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186903    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186906    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186909    2581 flags.go:64] FLAG: --v="2"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186914    2581 flags.go:64] FLAG: --version="false"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186918    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186924    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.186929    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187066    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187070    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187073    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187076    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187080    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187082    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187085    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187087    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187090    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187093    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.196074 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187096    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187098    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187101    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187103    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187107    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187111    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187114    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187117    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187121    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187124    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187126    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187129    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187132    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187134    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187137    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187140    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187142    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187145    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187148    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.196600 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187150    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187154    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187157    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187160    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187163    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187167    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187171    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187175    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187177    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187180    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187183    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187185    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187188    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187190    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187193    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187195    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187198    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187200    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187203    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.197046 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187206    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187208    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187212    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187215    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187218    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187220    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187223    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187226    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187229    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187231    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187234    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187268    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187273    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187276    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187280    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187283    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187286    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187290    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187293    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187295    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.197526 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187298    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187300    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187303    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187306    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187309    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187311    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187314    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187316    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187319    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187322    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187324    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187338    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187341    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187354    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187358    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187361    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187364    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187366    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187369    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.198223 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187372    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187375    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187377    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187380    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187383    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187385    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187388    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187390    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187394    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187397    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187399    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187402    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187405    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187408    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187411    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187413    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187416    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187419    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.198946 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187421    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.187450    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187487    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187491    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187495    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187497    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187500    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187503    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187505    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187508    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187512    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187515    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187518    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187521    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187523    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187526    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.199411 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187528    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187531    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187534    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187536    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187539    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187542    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187546    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187549    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187552    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187554    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187562    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187565    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187567    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187570    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187573    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187575    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187578    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187581    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187583    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.199790 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187586    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187588    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187591    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187594    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187596    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187599    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187602    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187606    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187609    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187612    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187614    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187617    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187620    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187622    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187625    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187628    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187630    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187633    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187636    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.200239 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187640    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187642    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187645    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187647    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187650    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187653    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187656    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187659    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187662    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187665    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187668    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187670    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187673    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187675    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187678    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187680    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187683    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187686    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187690    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187693    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.200727 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187698    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187701    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187704    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187707    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187709    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187712    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187715    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187718    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187720    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187723    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187726    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187728    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187732    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187735    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187738    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187740    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187742    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187745    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187749    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.201199 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187753    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187756    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187759    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187762    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187765    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187767    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187770    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187772    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187775    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187778    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187780    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187783    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187785    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187789    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.187793    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.195350    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.195363    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.195380    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.195384    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195460    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.201694 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195465    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195468    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195471    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195475    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195479    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195484    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195487    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195490    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195494    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195497    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195500    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195509    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195512    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195515    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195517    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195520    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195523    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195525    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195528    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.202203 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195531    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195534    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195536    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195539    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195542    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195544    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195548    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195550    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195553    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195556    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195559    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195561    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195564    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195566    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195569    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195572    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195574    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195577    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195579    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.202669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195582    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195585    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195587    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195590    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195592    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195595    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195603    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195606    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195608    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195611    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195613    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195616    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195618    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195621    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195623    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195626    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195628    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195631    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195633    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195636    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.203110 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195639    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195642    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195644    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195647    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195650    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195652    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195655    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195657    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195660    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195663    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195665    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195668    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195671    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195673    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195676    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195679    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195681    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195684    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195687    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195695    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.203599 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195697    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195700    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195703    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195705    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195708    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195710    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195713    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195715    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195718    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195720    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195724    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195726    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195729    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195731    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195734    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195738    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195742    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195745    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195748    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.204068 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195750    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195753    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195755    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195758    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195760    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195763    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195766    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.195768    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.195794    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196014    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196027    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196030    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196033    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196041    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196044    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196047    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.204516 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196050    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196052    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196055    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196058    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196060    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196063    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196065    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196068    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196071    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196073    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196076    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196079    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196082    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196084    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196087    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196091    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196096    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196098    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196101    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.204899 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196104    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196107    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196110    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196112    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196115    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196117    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196120    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196123    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196125    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196128    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196130    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196138    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196141    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196144    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196146    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196149    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196151    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196154    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196156    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196159    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.205477 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196161    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196164    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196167    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196169    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196172    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196174    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196177    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196179    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196182    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196185    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196187    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196190    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196192    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196195    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196197    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196200    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196202    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196205    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196208    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.205948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196210    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196212    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196215    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196218    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196220    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196228    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196231    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196234    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196236    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196239    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196242    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196244    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196246    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196249    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196252    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196254    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196256    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196259    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196261    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196264    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.206415 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196266    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196269    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196271    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196274    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196282    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196285    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196287    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196290    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196292    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196295    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196299    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196302    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196305    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196308    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196310    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196313    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196317    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196320    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196340    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.206878 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196343    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196346    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.196370    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196418    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196422    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196425    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196427    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196430    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196433    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196435    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196438    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196441    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196443    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196446    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196449    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196451    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:29:21.207324 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196454    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196456    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196459    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196461    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196464    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196467    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196471    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196475    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196478    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196481    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196484    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196486    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196489    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196492    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196496    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196499    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196508    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196511    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196513    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:29:21.207732 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196516    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196519    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196521    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196524    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196526    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196529    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196531    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196534    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196536    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196539    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196541    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196544    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196547    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196549    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196552    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196555    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196557    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196560    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196562    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:29:21.208176 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196565    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196567    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196570    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196573    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196575    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196578    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196580    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196582    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196586    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196589    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196591    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196599    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196602    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196605    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196608    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196610    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196613    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196616    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196618    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196621    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:29:21.208669 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196624    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196626    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196629    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196631    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196634    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196637    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196639    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196643    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196646    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196649    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196651    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196654    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196657    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196660    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196662    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196665    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196667    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196670    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196672    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196675    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:29:21.209142 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196679    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196682    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196686    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196689    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196698    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196701    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196704    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196706    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196709    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196712    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196714    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196717    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196720    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196723    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:21.196725    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.196775    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.200588    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.201657    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 11:29:21.209779 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.201752    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:29:21.210247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.201789    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:29:21.210247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.203961    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:21.210247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.209313    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:29:21.219135 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.219115    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 11:29:21.223165 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.223151    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 11:29:21.224047 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.224017    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 11:29:21.228339 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.228309    2581 fs.go:135] Filesystem UUIDs: map[6350d68a-01fe-420e-86ac-068375079e1d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a293865f-bd7e-4a36-8bf2-91af6c5653fb:/dev/nvme0n1p3]
Oct 30 11:29:21.228406 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.228344    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 11:29:21.234234 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.234109    2581 manager.go:217] Machine: {Timestamp:2026-10-30 11:29:21.232619197 +0000 UTC m=+0.362664858 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099458 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284400a8315eaaa8cb56454fcd5930 SystemUUID:ec284400-a831-5eaa-a8cb-56454fcd5930 BootID:da411877-6f5f-46f7-b701-858aedc60db9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:12:a2:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:12:a2:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:50:ba:29:f4:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 11:29:21.234234 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.234231    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 11:29:21.234358 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.234313    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 11:29:21.237850 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.237823    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 11:29:21.237994 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.237858    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 11:29:21.238035 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.237997    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 11:29:21.238035 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.238005    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 11:29:21.238092 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.238035    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:21.238092 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.238053    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:29:21.239325 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.239314    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 11:29:21.245680 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.245667    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 11:29:21.245799 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.245790    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 11:29:21.246642 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.246632    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 11:29:21.246686 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.246653    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 11:29:21.246686 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.246664    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 11:29:21.246686 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.246673    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 11:29:21.246790 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.246691    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 11:29:21.247281 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.247267    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:21.251464 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.251442    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 11:29:21.258358 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.258325    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:29:21.259237 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.259225    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 11:29:21.259278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.259256    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 11:29:21.260055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260042    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 11:29:21.260105 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260060    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 11:29:21.260105 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260066    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 11:29:21.260105 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260073    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 11:29:21.260105 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260078    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 11:29:21.260105 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260084    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260162    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260169    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260176    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260181    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260190    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 11:29:21.260228 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260199    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 11:29:21.260985 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260974    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 11:29:21.260985 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.260984    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 11:29:21.265035 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.265019    2581 server.go:1263] "Started kubelet"
Oct 30 11:29:21.265162 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.265109    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 11:29:21.265275 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.265189    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 11:29:21.265314 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.265300    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 11:29:21.265945 ip-10-0-143-18 systemd[1]: Started Kubernetes Kubelet.
Oct 30 11:29:21.266748 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.266728    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 11:29:21.266992 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.266975    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 11:29:21.271363 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.271300    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 11:29:21.275572 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.275548    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 11:29:21.276212 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.276190    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 11:29:21.276317 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.276291    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.277011 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.276991    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 11:29:21.277011 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277014    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 11:29:21.277169 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277036    2581 factory.go:55] Registering systemd factory
Oct 30 11:29:21.277169 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.276986    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 11:29:21.277169 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277081    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 11:29:21.277307 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277205    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 11:29:21.277307 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277214    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 11:29:21.277395 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277351    2581 factory.go:153] Registering CRI-O factory
Oct 30 11:29:21.277395 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277368    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 11:29:21.277447 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277411    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 11:29:21.277447 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277426    2581 factory.go:103] Registering Raw factory
Oct 30 11:29:21.277447 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.277436    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 11:29:21.278755 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.278726    2581 manager.go:319] Starting recovery of all containers
Oct 30 11:29:21.291125 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.291106    2581 manager.go:324] Recovery completed
Oct 30 11:29:21.295517 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.295499    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.297409 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.297383    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.297487 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.297423    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.297487 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.297434    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.298012 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.297999    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.298071 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.297998    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 11:29:21.298071 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.298026    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 11:29:21.298071 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.298047    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 11:29:21.298707 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.298690    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.298761 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.298723    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.298761 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.298737    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.300279 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.300262    2581 policy_none.go:50] "Start"
Oct 30 11:29:21.300370 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.300284    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 11:29:21.300370 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.300299    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 11:29:21.302397 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.302384    2581 policy_none.go:44] "Start"
Oct 30 11:29:21.302571 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.302556    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 11:29:21.320809 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.320651    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 11:29:21.320995 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.320973    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-18.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 11:29:21.321218 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.320865    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 11:29:21.323248 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.320616    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-18.ec2.internal.18e34b3384ade492  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-18.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-18.ec2.internal,},FirstTimestamp:2026-10-30 11:29:21.264993426 +0000 UTC m=+0.395039086,LastTimestamp:2026-10-30 11:29:21.264993426 +0000 UTC m=+0.395039086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-18.ec2.internal,}"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.332935    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.336904    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.336938    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.336919    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-18.ec2.internal.18e34b33869c8010  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-18.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-18.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-18.ec2.internal,},FirstTimestamp:2026-10-30 11:29:21.297408016 +0000 UTC m=+0.427453676,LastTimestamp:2026-10-30 11:29:21.297408016 +0000 UTC m=+0.427453676,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-18.ec2.internal,}"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339119    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.339144    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339155    2581 server.go:87] "Starting device plugin registration server"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339445    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339454    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339541    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339623    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.339631    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.340285    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.340378    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.345294    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.346530    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.346549    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.346567    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 11:29:21.353055 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.346599    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 11:29:21.355817 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.355014    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-18.ec2.internal.18e34b33869ccc73  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-18.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-18.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-18.ec2.internal,},FirstTimestamp:2026-10-30 11:29:21.297427571 +0000 UTC m=+0.427473231,LastTimestamp:2026-10-30 11:29:21.297427571 +0000 UTC m=+0.427473231,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-18.ec2.internal,}"
Oct 30 11:29:21.365787 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.365767    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j42m"
Oct 30 11:29:21.374311 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.374238    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-18.ec2.internal.18e34b33869cfc97  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-18.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-18.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-18.ec2.internal,},FirstTimestamp:2026-10-30 11:29:21.297439895 +0000 UTC m=+0.427485557,LastTimestamp:2026-10-30 11:29:21.297439895 +0000 UTC m=+0.427485557,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-18.ec2.internal,}"
Oct 30 11:29:21.378095 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.378079    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9j42m"
Oct 30 11:29:21.405405 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.405381    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:21.440361 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.440275    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.441384 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.441366    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.441472 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.441398    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.441472 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.441411    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.441472 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.441436    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.448120 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.448097    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"]
Oct 30 11:29:21.448206 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.448159    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.448964 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.448951    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.449023 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.448978    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.449023 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.448989    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.451170 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451157    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.451300 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.451355 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451316    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.451892 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451878    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.451950 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451898    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.451950 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451908    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.451950 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451921    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.452046 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451954    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.452046 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.451966    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.453973 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.453959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.454035 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.453983    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:21.454696 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.454674    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:21.454803 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.454704    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:21.454803 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.454718    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:21.460155 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.460132    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.460256 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.460170    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-18.ec2.internal\": node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.478140 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.478099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.478236 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.478144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.478236 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.478174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.481139 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.481121    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-18.ec2.internal\" not found" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.483917 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.483897    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-18.ec2.internal\" not found" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.499803 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.499779    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.578374 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.578374 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.578603 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.578603 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578434    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.578603 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.578603 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.578525    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.600490 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.600460    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.701275 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.701194    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.786386 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.786353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.789498 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:21.789483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:21.801376 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.801359    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:21.901692 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:21.901652    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:22.003206 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:22.003173    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:22.104051 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:22.104005    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:22.201954 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.201922    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 11:29:22.202531 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.202108    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 11:29:22.204276 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:22.204252    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:22.304412 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:22.304322    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:29:22.331870 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.331842    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:29:22.333264 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.333242    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:29:22.333382 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.333280    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:29:22.333382 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.333295    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:29:22.346330 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:22.346297    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cd7a0c462fc7ca3f08389f12997e13.slice/crio-406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765 WatchSource:0}: Error finding container 406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765: Status 404 returned error can't find the container with id 406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765
Oct 30 11:29:22.346864 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:22.346843    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6905fd60ff57bade2ab9f0792ffa70c2.slice/crio-5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b WatchSource:0}: Error finding container 5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b: Status 404 returned error can't find the container with id 5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b
Oct 30 11:29:22.351805 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.351783    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:29:22.383528 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.383491    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:22 +0000 UTC" deadline="2028-05-06 19:38:10.51894255 +0000 UTC"
Oct 30 11:29:22.383528 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.383521    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13304h8m48.135425148s"
Oct 30 11:29:22.390212 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.390188    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:22.479924 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.479877    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:22.508989 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.508966    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:22.511175 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.511160    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:29:22.533556 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.533532    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:29:22.711950 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.711876    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:22.728159 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:22.728128    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:23.069991 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.069912    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:29:23.247817 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.247784    2581 apiserver.go:52] "Watching apiserver"
Oct 30 11:29:23.277460 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.277417    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 11:29:23.280020 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.279994    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 11:29:23.280436 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.280395    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-ptd94","openshift-multus/multus-l5l6p","openshift-network-diagnostics/network-check-target-qq4wf","openshift-ovn-kubernetes/ovnkube-node-9gwhb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d","kube-system/global-pull-secret-syncer-hgkvl","openshift-cluster-node-tuning-operator/tuned-bkdnv","openshift-image-registry/node-ca-8bfdd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal","openshift-multus/network-metrics-daemon-wfrcp","kube-system/konnectivity-agent-6d7hf","kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal","openshift-dns/node-resolver-h6llc","openshift-multus/multus-additional-cni-plugins-d4wr8"]
Oct 30 11:29:23.284117 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.284083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.286530 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.286505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.289841 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.289502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:23.289841 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.289590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:23.292468 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.292447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.294617 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.294589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.297126 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.297013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.297126 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.297088    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:23.299562 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.299539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.301737 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.301715    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.301847 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.301744    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.301847 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.301823    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\""
Oct 30 11:29:23.302752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.302730    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.302972 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.302955    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.303064 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.303049    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\""
Oct 30 11:29:23.303374 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.303362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.305722 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.305701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.308973 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.308953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.309069 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.309023    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:23.311322 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.311302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.313975 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.313958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.338247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.338220    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 11:29:23.338345 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.338297    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8qc2"
Oct 30 11:29:23.351859 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.351834    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r8qc2"
Oct 30 11:29:23.355466 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.355416    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" event={"ID":"6905fd60ff57bade2ab9f0792ffa70c2","Type":"ContainerStarted","Data":"5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b"}
Oct 30 11:29:23.358552 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.358526    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerStarted","Data":"406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765"}
Oct 30 11:29:23.378604 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.378578    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 11:29:23.388822 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.388797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.388964 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.388833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvgx\" (UniqueName: \"kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.388964 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.388862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq46s\" (UniqueName: \"kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.388964 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.388931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389095 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.388977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.389095 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.389095 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.389247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.389247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389468 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389468 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmfg\" (UniqueName: \"kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389468 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.389468 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.389678 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389678 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389678 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389678 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.389678 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.389921 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.389921 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:23.389921 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389921 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.389921 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw9f\" (UniqueName: \"kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzk9\" (UniqueName: \"kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.389989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.390172 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp929\" (UniqueName: \"kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.390493 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjc6\" (UniqueName: \"kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.390827 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.390979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.391265 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zrb\" (UniqueName: \"kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwglp\" (UniqueName: \"kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.391752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.392384 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.392384 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.391770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410244    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410297    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410506    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410576    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410642    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.410753 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410713    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\""
Oct 30 11:29:23.411157 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.410987    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.411157 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411030    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.411157 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411072    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.411157 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411144    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411171    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411196    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411200    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411218    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411227    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411245    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411280    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411303    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411315    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 11:29:23.411356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.411319    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 11:29:23.412922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.412897    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 11:29:23.413217 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413195    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\""
Oct 30 11:29:23.413300 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413217    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 11:29:23.413300 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413281    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\""
Oct 30 11:29:23.413300 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413203    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 11:29:23.413300 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413284    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\""
Oct 30 11:29:23.413533 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413292    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 11:29:23.413533 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413340    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 11:29:23.413533 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.413377    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\""
Oct 30 11:29:23.425284 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.425256    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 11:29:23.492695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.492695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492842    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.492965 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.492989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw9f\" (UniqueName: \"kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzk9\" (UniqueName: \"kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493095    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.493278 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493296    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493365    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493554    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493557    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493614    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493632    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493684    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.493922 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493898    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493957    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.493957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp929\" (UniqueName: \"kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494046    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494108    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494152    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494190    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjc6\" (UniqueName: \"kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494213    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.494266 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494267    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494290    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494335    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494306    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494386    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494434    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.494896 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zrb\" (UniqueName: \"kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwglp\" (UniqueName: \"kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494984    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.494995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.495513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495059    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495165    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495187    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495217    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495253    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495277    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495430    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495511    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495875    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495958    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.496256 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496024    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496069    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496478    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496554    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496639    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.496999    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497065    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497113    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.497137 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497116    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497184    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.495311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497228    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvgx\" (UniqueName: \"kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq46s\" (UniqueName: \"kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.497695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmfg\" (UniqueName: \"kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497745    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497789    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497801    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497880    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498168    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498219    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498169    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498270    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.497818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.498359    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:23.498600 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.498447    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:23.998423871 +0000 UTC m=+3.128469539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498506    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.498609    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498641    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.498666    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:23.998650658 +0000 UTC m=+3.128696306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:23.499485 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.498723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.501742 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.501717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.501887 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.501848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.517218 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.517191    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:23.517218 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.517220    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:23.517481 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.517235    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:23.517481 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:23.517297    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:24.017278898 +0000 UTC m=+3.147324567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:23.520262 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.520230    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw9f\" (UniqueName: \"kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.520762 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.520739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzk9\" (UniqueName: \"kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.521696 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.521671    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjc6\" (UniqueName: \"kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:23.522011 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.521987    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp929\" (UniqueName: \"kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.523435 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.523041    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwglp\" (UniqueName: \"kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.527592 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.527520    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvgx\" (UniqueName: \"kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.527695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.527607    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq46s\" (UniqueName: \"kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.528130 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.528107    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmfg\" (UniqueName: \"kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.529306 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.529263    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zrb\" (UniqueName: \"kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.602578 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.602494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:29:23.610689 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.610657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:29:23.614808 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.614779    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaca0000_f0c2_4984_ac8b_558dfadf5a60.slice/crio-12f0eef9dff418bdaf364fac6e319cfc6d27dd6a47c15fc4349f6fd19bacf406 WatchSource:0}: Error finding container 12f0eef9dff418bdaf364fac6e319cfc6d27dd6a47c15fc4349f6fd19bacf406: Status 404 returned error can't find the container with id 12f0eef9dff418bdaf364fac6e319cfc6d27dd6a47c15fc4349f6fd19bacf406
Oct 30 11:29:23.622971 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.622939    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3a0a4d_2922_4948_af17_cf737a3f6273.slice/crio-ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8 WatchSource:0}: Error finding container ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8: Status 404 returned error can't find the container with id ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8
Oct 30 11:29:23.641811 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.641779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:23.643299 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.643255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:29:23.651762 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.651575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:29:23.653806 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.653785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5l6p"
Oct 30 11:29:23.664246 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.664213    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16817ce5_34ed_46e3_9969_f4d62ea2eea9.slice/crio-2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512 WatchSource:0}: Error finding container 2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512: Status 404 returned error can't find the container with id 2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512
Oct 30 11:29:23.666188 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.666086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:29:23.668052 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.668009    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:23.670545 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:23.670513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:29:23.671313 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.671285    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2970c8b3_5474_4885_95e7_9562083216e2.slice/crio-d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676 WatchSource:0}: Error finding container d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676: Status 404 returned error can't find the container with id d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676
Oct 30 11:29:23.685968 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.685935    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca0a506_f0f2_4037_b2bb_312e2532b938.slice/crio-19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea WatchSource:0}: Error finding container 19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea: Status 404 returned error can't find the container with id 19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea
Oct 30 11:29:23.686657 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:23.686578    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb76f4db_7292_45d7_8780_f5e14eedbfd3.slice/crio-a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03 WatchSource:0}: Error finding container a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03: Status 404 returned error can't find the container with id a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03
Oct 30 11:29:24.002957 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.002685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:24.003147 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.002869    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:24.003147 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.003041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:24.003147 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.003104    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:25.003084634 +0000 UTC m=+4.133130297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:24.003336 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.003187    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:24.003336 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.003254    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:25.003236872 +0000 UTC m=+4.133282535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:24.104237 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.104044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:24.108819 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.108789    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:24.108966 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.108828    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:24.108966 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.108852    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:24.108966 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.108938    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:25.108909612 +0000 UTC m=+4.238955270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:24.347683 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.347018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:24.347683 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:24.347162    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:24.355124 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.355049    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:23 +0000 UTC" deadline="2028-05-20 21:35:18.9940668 +0000 UTC"
Oct 30 11:29:24.355124 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.355123    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13642h5m54.638948446s"
Oct 30 11:29:24.365766 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.365729    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bfdd" event={"ID":"cb76f4db-7292-45d7-8780-f5e14eedbfd3","Type":"ContainerStarted","Data":"a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03"}
Oct 30 11:29:24.370356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.370287    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerStarted","Data":"19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea"}
Oct 30 11:29:24.372037 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.372009    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6d7hf" event={"ID":"44f9925c-7d11-4296-9593-7a747f8d5916","Type":"ContainerStarted","Data":"618193af9f2c4c8f8c0c2cc87026fea8ce98fb5f20409cf30eeb90d4e83b5730"}
Oct 30 11:29:24.375554 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.375525    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ptd94" event={"ID":"7cb183e6-c4c5-4b96-a2a0-12897b0572ca","Type":"ContainerStarted","Data":"9bf75660c31abcaee9d9ecc764c0c9a164c58cf10e1bf781eb3215723f69d252"}
Oct 30 11:29:24.383274 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.383241    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512"}
Oct 30 11:29:24.387856 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.387823    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" event={"ID":"6905fd60ff57bade2ab9f0792ffa70c2","Type":"ContainerStarted","Data":"459e07ecd005cfa3bc2be292bb8495d0624229eb860944d14cec48dcb0ff2080"}
Oct 30 11:29:24.391201 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.391170    2581 generic.go:356] "Generic (PLEG): container finished" podID="78cd7a0c462fc7ca3f08389f12997e13" containerID="4cc436a8634c85d634153fbd69eb8ae3cd68aa2b0daba46219214d7cb5457b62" exitCode=0
Oct 30 11:29:24.391318 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.391251    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerDied","Data":"4cc436a8634c85d634153fbd69eb8ae3cd68aa2b0daba46219214d7cb5457b62"}
Oct 30 11:29:24.395111 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.395064    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5l6p" event={"ID":"5727749f-b29a-44b7-8b7e-02b65d349ed6","Type":"ContainerStarted","Data":"ab39c7dcc480aaecf47134b8495ef5400cd7306ab6b12e6660466721e75e84d6"}
Oct 30 11:29:24.398196 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.398172    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d" event={"ID":"2970c8b3-5474-4885-95e7-9562083216e2","Type":"ContainerStarted","Data":"d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676"}
Oct 30 11:29:24.399583 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.399560    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6llc" event={"ID":"ef3a0a4d-2922-4948-af17-cf737a3f6273","Type":"ContainerStarted","Data":"ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8"}
Oct 30 11:29:24.407068 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.407037    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkdnv" event={"ID":"baca0000-f0c2-4984-ac8b-558dfadf5a60","Type":"ContainerStarted","Data":"12f0eef9dff418bdaf364fac6e319cfc6d27dd6a47c15fc4349f6fd19bacf406"}
Oct 30 11:29:24.434721 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:24.434383    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podStartSLOduration=1.434363734 podStartE2EDuration="1.434363734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:24.433040916 +0000 UTC m=+3.563086593" watchObservedRunningTime="2026-10-30 11:29:24.434363734 +0000 UTC m=+3.564409407"
Oct 30 11:29:25.018774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.018685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:25.018774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.018766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:25.019054 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.018938    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:25.019054 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.018999    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:27.018981257 +0000 UTC m=+6.149026921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:25.019173 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.019066    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:25.019173 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.019103    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:27.019091614 +0000 UTC m=+6.149137279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:25.119731 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.119695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:25.119914 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.119870    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:25.119914 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.119891    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:25.119914 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.119904    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:25.120078 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.119963    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:27.119940943 +0000 UTC m=+6.249986608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:25.351072 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.351002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:25.351538 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.351154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:25.351538 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.351238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:25.351538 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:25.351313    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:25.355295 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.355256    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:23 +0000 UTC" deadline="2028-04-27 07:04:57.411561683 +0000 UTC"
Oct 30 11:29:25.355295 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.355295    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13075h35m32.056272313s"
Oct 30 11:29:25.413126 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:25.413053    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerStarted","Data":"3c1fd474751bd7561c5108f608ee26b07e8e911dd98f7d73c2531d6daa168b2c"}
Oct 30 11:29:26.347394 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:26.347359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:26.347601 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:26.347506    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:27.035844 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.035796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:27.036344 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.035873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:27.036344 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.036042    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:27.036344 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.036111    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:31.036092698 +0000 UTC m=+10.166138372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:27.036605 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.036571    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:27.036663 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.036634    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:31.03661595 +0000 UTC m=+10.166661615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:27.136913 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.136821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:27.137084 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.137046    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:27.137084 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.137069    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:27.137190 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.137087    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:27.137190 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.137154    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:31.137134259 +0000 UTC m=+10.267179932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:27.349929 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.349899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:27.350124 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.350033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:27.350944 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.350437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:27.350944 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:27.350555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:27.426048 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.425948    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="508e57d0ca13b9f5c261c6638ca65b89d01378bfc96b3f4b4898df0e86d20eea" exitCode=0
Oct 30 11:29:27.426048 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.426007    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"508e57d0ca13b9f5c261c6638ca65b89d01378bfc96b3f4b4898df0e86d20eea"}
Oct 30 11:29:27.507578 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:27.506261    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" podStartSLOduration=4.506243087 podStartE2EDuration="4.506243087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:25.459674044 +0000 UTC m=+4.589719717" watchObservedRunningTime="2026-10-30 11:29:27.506243087 +0000 UTC m=+6.636288762"
Oct 30 11:29:28.348045 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:28.347537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:28.348045 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:28.347666    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:28.438465 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:28.437705    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6d7hf" event={"ID":"44f9925c-7d11-4296-9593-7a747f8d5916","Type":"ContainerStarted","Data":"f3aaf3557dd055f10849810ae907548e3afb8d58aad72f1d560542cfea4d5d1e"}
Oct 30 11:29:29.347699 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:29.347621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:29.347845 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:29.347782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:29.348220 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:29.348180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:29.348590 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:29.348275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:29.449112 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:29.449068    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8bfdd" event={"ID":"cb76f4db-7292-45d7-8780-f5e14eedbfd3","Type":"ContainerStarted","Data":"ac613ccdc281635e9ec479faf3c66fbb31c7fdd4a16b226fb7b20f30f4f90fb5"}
Oct 30 11:29:29.492931 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:29.491825    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6d7hf" podStartSLOduration=3.606508147 podStartE2EDuration="7.491803889s" totalImagesPullingTime="3.885295742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:28.483259522 +0000 UTC m=+7.613305196" watchObservedRunningTime="2026-10-30 11:29:29.491803889 +0000 UTC m=+8.621849567"
Oct 30 11:29:30.347832 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:30.347796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:30.348009 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:30.347956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:30.454118 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:30.454031    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d" event={"ID":"2970c8b3-5474-4885-95e7-9562083216e2","Type":"ContainerStarted","Data":"aaf6abe287dee3f2eae529c8c44bcbf67219340746102183ba0a7d07e57ceb24"}
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.075051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.075129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.075290    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.075370    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:39.075351434 +0000 UTC m=+18.205397100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.075812    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:31.075925 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.075872    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:39.07584799 +0000 UTC m=+18.205893655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:31.175880 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.175828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:31.176059 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.176009    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:31.176059 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.176031    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:31.176059 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.176043    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:31.176216 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.176103    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:39.176084483 +0000 UTC m=+18.306130150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:31.352399 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.350732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:31.352399 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.350812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:31.352399 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.351111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:31.352399 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:31.351196    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:31.456311 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.454890    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 11:29:31.459893 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:31.459810    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d" event={"ID":"2970c8b3-5474-4885-95e7-9562083216e2","Type":"ContainerStarted","Data":"979eb56ac12672442a52edba03f71ffe30cd3db12c7778fe19ad619feb022a8c"}
Oct 30 11:29:32.345305 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.344943    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:29:31.454908499Z","UUID":"96fbfc3e-ff88-4671-8cb2-1ac12202bc24","Handler":null,"Name":"","Endpoint":""}
Oct 30 11:29:32.346520 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.346499    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 11:29:32.346624 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.346524    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 11:29:32.346894 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.346680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:32.346894 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:32.346766    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:32.464236 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.464168    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5l6p" event={"ID":"5727749f-b29a-44b7-8b7e-02b65d349ed6","Type":"ContainerStarted","Data":"8e370b8fad5abf258cff684b036e3b83cb3bc16a3b1428b94186159696c71fc4"}
Oct 30 11:29:32.467368 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.467321    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d" event={"ID":"2970c8b3-5474-4885-95e7-9562083216e2","Type":"ContainerStarted","Data":"b5f8d7f7b3d6fb0cfeafb89986eddffaa51a4a4ef63918c4de94c4b414a11422"}
Oct 30 11:29:32.469709 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.469683    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h6llc" event={"ID":"ef3a0a4d-2922-4948-af17-cf737a3f6273","Type":"ContainerStarted","Data":"23e28b4c811a83415c01d444cf2bff7657550367e17602f98b4a3184db327308"}
Oct 30 11:29:32.472259 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.472235    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="4063de210c57fb2d8b118049d43462b3ab59428cbbe4d64cd734291a1d744d0c" exitCode=0
Oct 30 11:29:32.472383 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.472270    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"4063de210c57fb2d8b118049d43462b3ab59428cbbe4d64cd734291a1d744d0c"}
Oct 30 11:29:32.516545 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.516496    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8bfdd" podStartSLOduration=5.130188263 podStartE2EDuration="10.516480618s" totalImagesPullingTime="5.386292355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:29.492903771 +0000 UTC m=+8.622949444" watchObservedRunningTime="2026-10-30 11:29:32.516480618 +0000 UTC m=+11.646526289"
Oct 30 11:29:32.557186 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.557137    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-l5l6p" podStartSLOduration=2.709492169 podStartE2EDuration="10.557120639s" totalImagesPullingTime="7.84762847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:32.516128096 +0000 UTC m=+11.646173766" watchObservedRunningTime="2026-10-30 11:29:32.557120639 +0000 UTC m=+11.687166309"
Oct 30 11:29:32.557751 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.557724    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h6llc" podStartSLOduration=2.226151149 podStartE2EDuration="10.557719138s" totalImagesPullingTime="8.331567989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:32.556812359 +0000 UTC m=+11.686858027" watchObservedRunningTime="2026-10-30 11:29:32.557719138 +0000 UTC m=+11.687764807"
Oct 30 11:29:32.617369 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:32.617287    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d" podStartSLOduration=2.35873097 podStartE2EDuration="10.617267769s" totalImagesPullingTime="8.258536799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:32.609652329 +0000 UTC m=+11.739697999" watchObservedRunningTime="2026-10-30 11:29:32.617267769 +0000 UTC m=+11.747313438"
Oct 30 11:29:33.272211 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.272173    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:33.273050 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.273024    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:33.347129 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.347053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:33.347286 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:33.347190    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:33.347609 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.347569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:33.347677 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:33.347662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:33.476506 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.476457    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="e381d8eba4171a63c3be25c98828f19e038ccdaa578e05d04132bc9283afdda2" exitCode=0
Oct 30 11:29:33.477219 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.476563    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"e381d8eba4171a63c3be25c98828f19e038ccdaa578e05d04132bc9283afdda2"}
Oct 30 11:29:33.478603 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.478574    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ptd94" event={"ID":"7cb183e6-c4c5-4b96-a2a0-12897b0572ca","Type":"ContainerStarted","Data":"aa47da1a37bda10b0058d104c5906b034d573e8737d317f63badb8e2b75e627a"}
Oct 30 11:29:33.564234 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:33.564177    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ptd94" podStartSLOduration=3.5370702510000003 podStartE2EDuration="11.564157234s" totalImagesPullingTime="8.027086983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:33.564039027 +0000 UTC m=+12.694084701" watchObservedRunningTime="2026-10-30 11:29:33.564157234 +0000 UTC m=+12.694202904"
Oct 30 11:29:34.347297 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:34.347270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:34.347419 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:34.347396    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:34.481943 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:34.481905    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkdnv" event={"ID":"baca0000-f0c2-4984-ac8b-558dfadf5a60","Type":"ContainerStarted","Data":"530e1d20deb72abf2fae16d0475adee25d40c3186b9f4653a1a6f104bd9682a9"}
Oct 30 11:29:34.526411 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:34.526316    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkdnv" podStartSLOduration=1.8813444540000002 podStartE2EDuration="12.526304462s" totalImagesPullingTime="10.644960008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:34.526165784 +0000 UTC m=+13.656211464" watchObservedRunningTime="2026-10-30 11:29:34.526304462 +0000 UTC m=+13.656350131"
Oct 30 11:29:35.347774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:35.347313    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:35.347774 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:35.347449    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:35.347774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:35.347498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:35.347774 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:35.347598    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:35.486578 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:35.486540    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="45aa41903563e332519e982dd552d864a3ad85dcfcfb08b134fdcafb84c0a1a2" exitCode=0
Oct 30 11:29:35.487106 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:35.486624    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"45aa41903563e332519e982dd552d864a3ad85dcfcfb08b134fdcafb84c0a1a2"}
Oct 30 11:29:36.347968 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:36.347728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:36.348153 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:36.348082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:37.347247 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:37.347165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:37.347653 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:37.347320    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:37.347653 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:37.347373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:37.347653 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:37.347502    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:37.494977 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:37.494931    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="4f9d5cd1edc27e181a50f2ea6dd0da93ab0703caa862a23af43e6340199bcfc7" exitCode=0
Oct 30 11:29:37.495130 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:37.494994    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"4f9d5cd1edc27e181a50f2ea6dd0da93ab0703caa862a23af43e6340199bcfc7"}
Oct 30 11:29:38.346961 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:38.346934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:38.347121 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:38.347026    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:38.499523 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:38.499491    2581 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="5471475adefd53bf85dac5f2e2d95e7a5c2c8ac9b5687ac4ba19a61082ece593" exitCode=0
Oct 30 11:29:38.499894 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:38.499544    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerDied","Data":"5471475adefd53bf85dac5f2e2d95e7a5c2c8ac9b5687ac4ba19a61082ece593"}
Oct 30 11:29:39.143954 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.143875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:39.143954 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.143922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:39.144126 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.144036    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:39.144126 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.144085    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret podName:2123026e-ac5a-4ef1-80f6-76b1bcc174c1 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.144072698 +0000 UTC m=+34.274118359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret") pod "global-pull-secret-syncer-hgkvl" (UID: "2123026e-ac5a-4ef1-80f6-76b1bcc174c1") : object "kube-system"/"original-pull-secret" not registered
Oct 30 11:29:39.144126 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.144037    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:39.144240 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.144157    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.144144357 +0000 UTC m=+34.274190004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 11:29:39.244576 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.244541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:39.244732 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.244692    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 11:29:39.244732 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.244713    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 11:29:39.244732 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.244722    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2vktr for pod openshift-network-diagnostics/network-check-target-qq4wf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:39.244827 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.244770    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr podName:9e0648c5-c6b4-49db-9dd2-4f058e911679 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.244757717 +0000 UTC m=+34.374803364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2vktr" (UniqueName: "kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr") pod "network-check-target-qq4wf" (UID: "9e0648c5-c6b4-49db-9dd2-4f058e911679") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 11:29:39.269118 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.269094    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:39.269282 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.269200    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:39.269623 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.269611    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:29:39.347356 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.347318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:39.347496 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.347425    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:39.347537 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.347507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:39.347613 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:39.347592    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:39.505132 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.505052    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" event={"ID":"aca0a506-f0f2-4037-b2bb-312e2532b938","Type":"ContainerStarted","Data":"f595f9fd80e5732cb9ec85d1ab9155f362a395cd9e3cd84a8c25e6ca9eb38505"}
Oct 30 11:29:39.507942 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.507912    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"a4b36af13b4c2ee060269e46585d27512e55394ae50f30c9f15cf6d4a87464b0"}
Oct 30 11:29:39.508074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.507979    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"4b38a4c677a28b97107117a72a3f4be91e082b5305c831d07de30ea2b9c4b208"}
Oct 30 11:29:39.508074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.508010    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"0af8743ad9c3088ad7d70e97cb8d95d3008d5af7f02d77d577df147a977796a0"}
Oct 30 11:29:39.508074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.508042    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"9fc86d51212a2905b43210008a5c32a1cf018e7b07cb46e1fa18aece4fa10db8"}
Oct 30 11:29:39.508074 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.508069    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"8974fd8209a3f75f02195f8b4072e64a18b19477b79a2dce27e6daba15ec0e31"}
Oct 30 11:29:39.508205 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.508096    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"a64e796b02e409aaf0f6d13b91dec52d74376f890fde4b20ec2ad3fd6241ea24"}
Oct 30 11:29:39.553267 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:39.553221    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d4wr8" podStartSLOduration=6.918853376 podStartE2EDuration="17.553208272s" totalImagesPullingTime="10.634354896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:39.552661472 +0000 UTC m=+18.682707142" watchObservedRunningTime="2026-10-30 11:29:39.553208272 +0000 UTC m=+18.683253941"
Oct 30 11:29:40.346758 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:40.346712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:40.346915 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:40.346829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:41.347609 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:41.347525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:41.348141 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:41.347622    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:41.348141 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:41.347735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:41.348141 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:41.347836    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:41.515161 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:41.515130    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"3110bbce8efae1a79a5d5a626d15a0d1602ac38dd7797e1446e92f7fa33113ae"}
Oct 30 11:29:42.347479 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:42.347453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:42.347649 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:42.347550    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:43.350978 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.350801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:43.351572 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.350894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:43.351572 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:43.351080    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:43.351572 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:43.351154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:43.522489 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.522406    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" event={"ID":"16817ce5-34ed-46e3-9969-f4d62ea2eea9","Type":"ContainerStarted","Data":"dd78ea8681b87171304ba68b1bc8d5ad9ceab115d1896d8629462d31b5082288"}
Oct 30 11:29:43.522794 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.522766    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:43.522920 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.522836    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:43.538107 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.538087    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:43.538283 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.538212    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:43.615491 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:43.615450    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb" podStartSLOduration=6.550067351 podStartE2EDuration="21.615438236s" totalImagesPullingTime="15.065370885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:43.614748402 +0000 UTC m=+22.744794071" watchObservedRunningTime="2026-10-30 11:29:43.615438236 +0000 UTC m=+22.745483915"
Oct 30 11:29:44.347720 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:44.347695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:44.347853 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:44.347781    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:44.525364 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:44.525320    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:45.327656 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.327563    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgkvl"]
Oct 30 11:29:45.328035 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.327999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:45.328166 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:45.328140    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:45.329114 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.329090    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq4wf"]
Oct 30 11:29:45.329200 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.329177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:45.329264 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:45.329245    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:45.339158 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.339134    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfrcp"]
Oct 30 11:29:45.339244 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.339223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:45.339307 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:45.339292    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:45.527632 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:45.527600    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:47.347373 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:47.347326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:47.347813 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:47.347386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:47.347813 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:47.347351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:47.347813 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:47.347476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:47.347813 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:47.347574    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:47.347813 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:47.347667    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:49.346955 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:49.346921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:49.347557 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:49.346929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:49.347557 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:49.347017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:49.347557 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:49.347032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:49.347557 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:49.347106    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:49.347557 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:49.347236    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:50.769752 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:50.769716    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:50.770163 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:50.769962    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:29:50.781188 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:50.781165    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:29:51.347865 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:51.347686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:51.348025 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:51.347734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:51.348025 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:51.347941    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgkvl" podUID="2123026e-ac5a-4ef1-80f6-76b1bcc174c1"
Oct 30 11:29:51.348148 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:51.347761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:51.348148 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:51.348035    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq4wf" podUID="9e0648c5-c6b4-49db-9dd2-4f058e911679"
Oct 30 11:29:51.348250 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:51.348153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wfrcp" podUID="f7270d2a-df57-495e-82f2-0f34335e48e0"
Oct 30 11:29:52.137258 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.137228    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeReady"
Oct 30 11:29:52.137634 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.137385    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 11:29:52.196994 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.196928    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lvpqd"]
Oct 30 11:29:52.263970 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.263943    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fv5lx"]
Oct 30 11:29:52.264131 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.264080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.275271 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.275250    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d5gjj"]
Oct 30 11:29:52.275393 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.275378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.281741 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.281720    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 11:29:52.281838 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.281820    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\""
Oct 30 11:29:52.281888 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.281872    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 11:29:52.281928 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.281913    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 11:29:52.291342 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.291310    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvpqd"]
Oct 30 11:29:52.291415 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.291359    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fv5lx"]
Oct 30 11:29:52.291415 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.291382    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d5gjj"]
Oct 30 11:29:52.291472 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.291446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.335632 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.335609    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fjgw"]
Oct 30 11:29:52.347640 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.347621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.350499 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljc8\" (UniqueName: \"kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.350597 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.350597 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.350597 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.350597 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.350754 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.350806 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bk5\" (UniqueName: \"kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.350858 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.350912 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6kn\" (UniqueName: \"kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.350912 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.350987 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.350924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.359842 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.359819    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fjgw"]
Oct 30 11:29:52.380596 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380557    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 11:29:52.380596 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380590    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 11:29:52.380774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380633    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 11:29:52.380774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380656    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 11:29:52.380774 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380689    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380786    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380789    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380820    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380854    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380867    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380878    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 11:29:52.380919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380915    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qxb8z\""
Oct 30 11:29:52.381125 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.380919    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\""
Oct 30 11:29:52.451695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljc8\" (UniqueName: \"kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.451695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.451695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.451695 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451791    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bk5\" (UniqueName: \"kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451823    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls podName:38b69f25-5947-4982-8a04-bc09d51a87b3 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.951836002 +0000 UTC m=+32.081881663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls") pod "dns-default-d5gjj" (UID: "38b69f25-5947-4982-8a04-bc09d51a87b3") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451904    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.951885685 +0000 UTC m=+32.081931343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451912    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6kn\" (UniqueName: \"kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.451966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.451976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:52.951960314 +0000 UTC m=+32.082005980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.452010 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452036    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g6dn\" (UniqueName: \"kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452133    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452143    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.452544 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.452450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.455639 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.455622    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.476283 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.476256    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bk5\" (UniqueName: \"kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.476492 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.476470    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljc8\" (UniqueName: \"kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.476545 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.476529    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6kn\" (UniqueName: \"kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.552847 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.552816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6dn\" (UniqueName: \"kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.553002 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.552916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.553002 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.552971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.553628 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.553604    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.555169 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.555145    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.588041 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.588020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:29:52.599159 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.599138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6dn\" (UniqueName: \"kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.684948 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.684919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:29:52.956061 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.955979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:52.956061 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.956033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.956259 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.956065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:52.956259 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956187    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.956356 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956264    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls podName:38b69f25-5947-4982-8a04-bc09d51a87b3 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:53.956242158 +0000 UTC m=+33.086287812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls") pod "dns-default-d5gjj" (UID: "38b69f25-5947-4982-8a04-bc09d51a87b3") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:52.956356 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956185    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.956356 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956318    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:53.956306261 +0000 UTC m=+33.086351927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:52.956356 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956194    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.956488 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:52.956407    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:53.956395179 +0000 UTC m=+33.086440835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:52.957067 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:52.957049    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvpqd"]
Oct 30 11:29:52.959297 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:52.959279    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e3a31c_a58f_42b7_a29d_3f41ef0d3435.slice/crio-706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94 WatchSource:0}: Error finding container 706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94: Status 404 returned error can't find the container with id 706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94
Oct 30 11:29:53.007919 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.007901    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4fjgw"]
Oct 30 11:29:53.008858 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:53.008841    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd6fb8f_85c3_4a50_80b7_fca2fdc580a3.slice/crio-a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc WatchSource:0}: Error finding container a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc: Status 404 returned error can't find the container with id a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc
Oct 30 11:29:53.347216 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.347133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:53.347216 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.347164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:53.347684 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.347474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:53.363719 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.363700    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 11:29:53.363850 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.363834    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 11:29:53.364232 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.364216    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 11:29:53.364766 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.364746    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\""
Oct 30 11:29:53.364766 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.364753    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\""
Oct 30 11:29:53.364885 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.364754    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 11:29:53.364885 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.364816    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\""
Oct 30 11:29:53.545823 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.545783    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw" event={"ID":"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3","Type":"ContainerStarted","Data":"a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc"}
Oct 30 11:29:53.548384 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.548310    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvpqd" event={"ID":"96e3a31c-a58f-42b7-a29d-3f41ef0d3435","Type":"ContainerStarted","Data":"706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94"}
Oct 30 11:29:53.967203 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.967161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:53.967422 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.967220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:53.967422 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:53.967266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:53.967422 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967363    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:53.967422 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967414    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.96740154 +0000 UTC m=+35.097447188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:53.967760 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967730    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:53.967877 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967792    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls podName:38b69f25-5947-4982-8a04-bc09d51a87b3 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.967774482 +0000 UTC m=+35.097820144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls") pod "dns-default-d5gjj" (UID: "38b69f25-5947-4982-8a04-bc09d51a87b3") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:53.967877 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967728    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:53.967877 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:53.967833    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:55.967823646 +0000 UTC m=+35.097869307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:55.175784 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.175748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:29:55.176254 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.175813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:55.176254 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.175945    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 11:29:55.176254 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.176028    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs podName:f7270d2a-df57-495e-82f2-0f34335e48e0 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:27.176008379 +0000 UTC m=+66.306054032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs") pod "network-metrics-daemon-wfrcp" (UID: "f7270d2a-df57-495e-82f2-0f34335e48e0") : secret "metrics-daemon-secret" not found
Oct 30 11:29:55.178662 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.178637    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:55.180613 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.180592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:29:55.276916 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.276889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:55.279596 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.279569    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:55.503769 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.503742    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgkvl"]
Oct 30 11:29:55.504970 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:55.504946    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123026e_ac5a_4ef1_80f6_76b1bcc174c1.slice/crio-26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e WatchSource:0}: Error finding container 26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e: Status 404 returned error can't find the container with id 26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e
Oct 30 11:29:55.519520 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.519498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:55.554511 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.554470    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgkvl" event={"ID":"2123026e-ac5a-4ef1-80f6-76b1bcc174c1","Type":"ContainerStarted","Data":"26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e"}
Oct 30 11:29:55.555999 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.555964    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw" event={"ID":"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3","Type":"ContainerStarted","Data":"4bde99249715666e6087a249eed9ea878e26c2f21441ba1d7214ea1a947c0aa2"}
Oct 30 11:29:55.820046 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.820002    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw" podStartSLOduration=1.491074131 podStartE2EDuration="3.819986967s" totalImagesPullingTime="2.328912836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:55.608899328 +0000 UTC m=+34.738944999" watchObservedRunningTime="2026-10-30 11:29:55.819986967 +0000 UTC m=+34.950032636"
Oct 30 11:29:55.820628 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.820612    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq4wf"]
Oct 30 11:29:55.822948 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:29:55.822925    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0648c5_c6b4_49db_9dd2_4f058e911679.slice/crio-9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74 WatchSource:0}: Error finding container 9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74: Status 404 returned error can't find the container with id 9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74
Oct 30 11:29:55.984633 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.984592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:29:55.984783 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.984646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:55.984783 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:55.984681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:29:55.984859 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.984817    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 11:29:55.984906 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.984882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.984864252 +0000 UTC m=+39.114909916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : configmap "kube-rbac-proxy" not found
Oct 30 11:29:55.985302 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.985286    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 11:29:55.985368 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.985355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls podName:38b69f25-5947-4982-8a04-bc09d51a87b3 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.985322977 +0000 UTC m=+39.115368639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls") pod "dns-default-d5gjj" (UID: "38b69f25-5947-4982-8a04-bc09d51a87b3") : secret "dns-default-metrics-tls" not found
Oct 30 11:29:55.985430 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.985419    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:55.985466 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:29:55.985458    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:29:59.985446884 +0000 UTC m=+39.115492544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:29:56.320847 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.320788    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d7hf_44f9925c-7d11-4296-9593-7a747f8d5916/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:56.494949 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.494922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6llc_ef3a0a4d-2922-4948-af17-cf737a3f6273/dns-node-resolver/0.log"
Oct 30 11:29:56.571937 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.571841    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvpqd" event={"ID":"96e3a31c-a58f-42b7-a29d-3f41ef0d3435","Type":"ContainerStarted","Data":"9d93a85fe27162f43e8b5b88840d39b0b317e22a75523374b05087f3b7f30c0f"}
Oct 30 11:29:56.574029 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.573866    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq4wf" event={"ID":"9e0648c5-c6b4-49db-9dd2-4f058e911679","Type":"ContainerStarted","Data":"9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74"}
Oct 30 11:29:56.578675 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.578648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d7hf_44f9925c-7d11-4296-9593-7a747f8d5916/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:56.627297 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.627236    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lvpqd" podStartSLOduration=1.7540826790000001 podStartE2EDuration="4.627218417s" totalImagesPullingTime="2.873135738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:56.625808745 +0000 UTC m=+35.755854414" watchObservedRunningTime="2026-10-30 11:29:56.627218417 +0000 UTC m=+35.757264087"
Oct 30 11:29:56.763075 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.763044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d7hf_44f9925c-7d11-4296-9593-7a747f8d5916/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:56.960656 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:56.960601    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6d7hf_44f9925c-7d11-4296-9593-7a747f8d5916/konnectivity-agent/0.log" limit=1024
Oct 30 11:29:57.093473 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:57.093446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bfdd_cb76f4db-7292-45d7-8780-f5e14eedbfd3/node-ca/0.log"
Oct 30 11:29:57.894201 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:57.893497    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvpqd_96e3a31c-a58f-42b7-a29d-3f41ef0d3435/serve-healthcheck-canary/0.log"
Oct 30 11:29:59.584066 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:59.584029    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq4wf" event={"ID":"9e0648c5-c6b4-49db-9dd2-4f058e911679","Type":"ContainerStarted","Data":"330fa08364c217af7968fa8a7dc40f58cda25a577e85c06c9a750ef814345774"}
Oct 30 11:29:59.584513 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:59.584170    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:29:59.626490 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:29:59.626440    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qq4wf" podStartSLOduration=34.056647999 podStartE2EDuration="37.626422589s" totalImagesPullingTime="3.56977459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:29:59.625689447 +0000 UTC m=+38.755735117" watchObservedRunningTime="2026-10-30 11:29:59.626422589 +0000 UTC m=+38.756468257"
Oct 30 11:30:00.020043 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.019999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:00.020043 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.020056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:00.020319 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.020099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:00.020319 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:30:00.020246    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:00.020319 ip-10-0-143-18 kubenswrapper[2581]: E1030 11:30:00.020313    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:08.02029669 +0000 UTC m=+47.150342360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:00.020704 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.020674    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:00.022506 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.022483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:00.169569 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.169536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:00.508066 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.508036    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d5gjj"]
Oct 30 11:30:00.508240 ip-10-0-143-18 kubenswrapper[2581]: W1030 11:30:00.508215    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b69f25_5947_4982_8a04_bc09d51a87b3.slice/crio-6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3 WatchSource:0}: Error finding container 6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3: Status 404 returned error can't find the container with id 6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3
Oct 30 11:30:00.587949 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.587912    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d5gjj" event={"ID":"38b69f25-5947-4982-8a04-bc09d51a87b3","Type":"ContainerStarted","Data":"6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3"}
Oct 30 11:30:00.589631 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.589598    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgkvl" event={"ID":"2123026e-ac5a-4ef1-80f6-76b1bcc174c1","Type":"ContainerStarted","Data":"ee9150f8be0a16045fce8978b2f8cc9f6860e1eca8ea85048da4cb2a30ae6264"}
Oct 30 11:30:00.633530 ip-10-0-143-18 kubenswrapper[2581]: I1030 11:30:00.633471    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hgkvl" podStartSLOduration=33.7215239 podStartE2EDuration="38.633453619s" totalImagesPullingTime="4.911929719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 11:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:00.633301846 +0000 UTC m=+39.763347516" watchObservedRunningTime="2026-10-30 11:30:00.633453619 +0000 UTC m=+39.763499288"
Oct 30 11:30:00.724970 ip-10-0-143-18 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 11:30:00.740762 ip-10-0-143-18 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 11:30:00.740928 ip-10-0-143-18 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 11:30:00.740978 ip-10-0-143-18 systemd[1]: kubelet.service: Consumed 2.326s CPU time.
Oct 30 11:30:00.750030 ip-10-0-143-18 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 11:30:00.824897 ip-10-0-143-18 kubenswrapper[4453]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:30:00.824897 ip-10-0-143-18 kubenswrapper[4453]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 11:30:00.824897 ip-10-0-143-18 kubenswrapper[4453]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:30:00.824897 ip-10-0-143-18 kubenswrapper[4453]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 11:30:00.827002 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.826923    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.827002 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827001    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827006    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827010    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827014    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827017    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827019    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827022    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827025    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827028    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827030    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827033    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827036    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827038    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827046    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827049    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827052    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827055    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827057    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827060    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.827071 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827063    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827087    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827090    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827093    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827096    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827099    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827110    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827113    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827116    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827119    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827122    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827124    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827127    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827130    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827132    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827135    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827138    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827140    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827143    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.827579 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827146    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827148    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827151    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827154    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827157    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827159    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827162    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827165    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827169    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827171    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827174    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827177    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827180    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827182    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827185    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827189    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827192    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827194    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827197    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827200    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.828058 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827203    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827205    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827208    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827211    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827213    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827216    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827218    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827221    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827225    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827227    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827230    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827233    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827235    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827238    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827244    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827249    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827253    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827256    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827259    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827261    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.828580 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827264    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827267    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827270    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827273    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827277    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827281    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827284    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827287    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827290    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827293    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827295    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827298    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827300    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827303    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827306    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827309    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827311    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827314    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827317    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827320    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.829075 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827322    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827325    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827341    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827345    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827350    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827355    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.827360    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828127    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828134    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828138    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828141    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828143    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828146    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828149    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828152    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828155    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828158    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828160    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828163    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.829581 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828167    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828170    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828172    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828175    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828178    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828181    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828183    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828186    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828188    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828191    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828193    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828196    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828199    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828201    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828204    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828206    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828209    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828212    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828214    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828217    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.830066 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828219    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828223    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828226    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828229    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828231    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828234    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828237    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828240    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828243    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828245    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828248    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828251    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828253    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828256    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828258    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828261    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828264    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828266    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828269    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828271    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.830610 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828274    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828277    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828279    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828282    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828284    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828287    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828289    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828292    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828295    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828298    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828301    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828303    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828306    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828309    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828312    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828314    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828318    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828322    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828326    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.831105 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828346    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828350    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828354    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828360    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828364    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828367    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828370    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828372    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828375    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828378    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828380    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828383    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828386    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828389    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828391    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828394    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828397    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828399    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828402    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.831597 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828404    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828407    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828410    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828413    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828416    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828418    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828421    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828424    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828426    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828430    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828432    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828435    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828438    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828441    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828443    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828448    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828924    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828929    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828932    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828935    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.832069 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828938    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828941    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828944    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828946    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828949    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828951    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828954    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828957    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828959    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828962    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828965    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828968    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828970    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828973    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828976    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828979    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828981    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828984    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828990    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828992    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.832571 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828996    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.828999    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829002    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829004    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829007    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829010    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829012    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829015    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829018    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829020    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829023    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829026    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829028    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829031    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829034    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829036    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829039    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829042    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829045    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829047    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.833089 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829050    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829053    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829055    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829058    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829060    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829063    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829065    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829069    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829072    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829075    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829078    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829081    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829084    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829087    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829090    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829094    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829096    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829099    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829101    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829104    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.833981 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829107    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829109    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829112    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829115    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829117    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829120    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829123    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829125    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829128    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829131    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829133    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829136    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829138    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829141    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829143    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829146    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829150    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829153    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829156    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.834638 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829160    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829164    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829168    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829171    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829175    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829177    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829180    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829183    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829185    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829188    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829190    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829193    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829195    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829198    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829200    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829203    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829205    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829208    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.835122 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829210    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829213    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829215    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829218    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.829220    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829310    4453 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829317    4453 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829339    4453 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829346    4453 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829353    4453 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829358    4453 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829365    4453 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829369    4453 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829372    4453 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829377    4453 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829381    4453 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829384    4453 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829388    4453 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829391    4453 flags.go:64] FLAG: --cgroup-root=""
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829395    4453 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829398    4453 flags.go:64] FLAG: --client-ca-file=""
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829401    4453 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829404    4453 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 11:30:00.835586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829408    4453 flags.go:64] FLAG: --cluster-domain=""
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829411    4453 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829415    4453 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829418    4453 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829421    4453 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829443    4453 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829447    4453 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829450    4453 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829454    4453 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829457    4453 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829460    4453 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829463    4453 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829466    4453 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829469    4453 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829473    4453 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829477    4453 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829480    4453 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829483    4453 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829486    4453 flags.go:64] FLAG: --enable-server="true"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829489    4453 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829493    4453 flags.go:64] FLAG: --event-burst="100"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829496    4453 flags.go:64] FLAG: --event-qps="50"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829500    4453 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829505    4453 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 11:30:00.836138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829509    4453 flags.go:64] FLAG: --eviction-hard=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829513    4453 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829517    4453 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829520    4453 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829523    4453 flags.go:64] FLAG: --eviction-soft=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829526    4453 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829529    4453 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829532    4453 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829535    4453 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829539    4453 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829542    4453 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829545    4453 flags.go:64] FLAG: --feature-gates=""
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829549    4453 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829552    4453 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829556    4453 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829559    4453 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829562    4453 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829565    4453 flags.go:64] FLAG: --help="false"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829568    4453 flags.go:64] FLAG: --hostname-override="ip-10-0-143-18.ec2.internal"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829572    4453 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829575    4453 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829578    4453 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829581    4453 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829585    4453 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 11:30:00.836805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829588    4453 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829591    4453 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829594    4453 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829597    4453 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829600    4453 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829603    4453 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829607    4453 flags.go:64] FLAG: --kube-reserved=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829611    4453 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829614    4453 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829618    4453 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829620    4453 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829623    4453 flags.go:64] FLAG: --lock-file=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829627    4453 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829630    4453 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829633    4453 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829639    4453 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829642    4453 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829645    4453 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829649    4453 flags.go:64] FLAG: --logging-format="text"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829652    4453 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829655    4453 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829658    4453 flags.go:64] FLAG: --manifest-url=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829661    4453 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829665    4453 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829668    4453 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829672    4453 flags.go:64] FLAG: --max-pods="110"
Oct 30 11:30:00.837416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829675    4453 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829678    4453 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829682    4453 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829684    4453 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829687    4453 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829690    4453 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829694    4453 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829700    4453 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829703    4453 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829706    4453 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829709    4453 flags.go:64] FLAG: --pod-cidr=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829712    4453 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829716    4453 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829721    4453 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829724    4453 flags.go:64] FLAG: --port="10250"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829730    4453 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829734    4453 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dbe10fdb81214965"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829737    4453 flags.go:64] FLAG: --qos-reserved=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829740    4453 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829743    4453 flags.go:64] FLAG: --register-node="true"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829746    4453 flags.go:64] FLAG: --register-with-taints=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829750    4453 flags.go:64] FLAG: --registry-burst="10"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829753    4453 flags.go:64] FLAG: --registry-qps="5"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829756    4453 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829759    4453 flags.go:64] FLAG: --reserved-memory=""
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829764    4453 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 11:30:00.838165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829767    4453 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829770    4453 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829773    4453 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829776    4453 flags.go:64] FLAG: --runonce="false"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829779    4453 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829782    4453 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829786    4453 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829788    4453 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829792    4453 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829795    4453 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829798    4453 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829801    4453 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829804    4453 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829807    4453 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829810    4453 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829813    4453 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829816    4453 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829819    4453 flags.go:64] FLAG: --system-cgroups=""
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829824    4453 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829837    4453 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829840    4453 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829843    4453 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829847    4453 flags.go:64] FLAG: --tls-min-version=""
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829850    4453 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829853    4453 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 11:30:00.838822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829856    4453 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829859    4453 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829862    4453 flags.go:64] FLAG: --v="2"
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829866    4453 flags.go:64] FLAG: --version="false"
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829871    4453 flags.go:64] FLAG: --vmodule=""
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829875    4453 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.829878    4453 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830005    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830008    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830011    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830014    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830017    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830020    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830022    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830025    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830027    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830030    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830032    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830035    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830038    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830040    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.839543 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830043    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830046    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830048    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830051    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830055    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830059    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830061    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830064    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830067    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830070    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830073    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830075    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830078    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830081    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830084    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830086    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830089    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830092    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830094    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830097    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.840093 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830100    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830102    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830105    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830108    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830111    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830113    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830116    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830118    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830121    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830124    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830126    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830129    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830132    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830134    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830137    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830139    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830144    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830147    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830150    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.840608 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830153    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830155    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830158    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830160    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830163    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830166    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830168    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830171    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830174    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830176    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830179    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830183    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830186    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830189    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830192    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830195    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830198    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830201    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830203    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.841185 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830206    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830209    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830212    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830215    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830217    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830220    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830223    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830225    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830228    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830230    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830234    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830238    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830241    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830244    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830247    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830249    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830252    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830254    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830257    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830260    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.841735 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830262    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830265    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830268    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830270    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830273    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830276    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830278    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830281    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830284    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830286    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830289    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830292    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830294    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830298    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.830325    4453 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830372    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.842431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830377    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830380    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830384    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830386    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830389    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830391    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830394    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830400    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830402    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830405    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830408    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830410    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830413    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830416    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830418    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830421    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830423    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830426    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830428    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830431    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.843186 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830434    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830436    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830439    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830442    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830448    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830451    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830454    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830456    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830459    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830462    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830465    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830469    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830472    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830475    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830478    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830480    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830483    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830485    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830488    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830493    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.843850 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830496    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830498    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830501    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830504    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830506    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830509    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830511    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830514    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830517    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830519    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830522    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830524    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830527    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830529    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830532    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830535    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830537    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830540    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830543    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830546    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.844460 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830548    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830551    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830553    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830556    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830559    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830561    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830564    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830566    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830569    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830572    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830574    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830579    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830582    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830585    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830587    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830590    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830593    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830595    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830598    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.845017 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830600    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830603    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830606    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830608    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830611    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830614    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830616    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830619    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830622    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830625    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830628    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830631    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830634    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830637    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830639    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830642    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830646    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830649    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830652    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830655    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.845519 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830657    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830660    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830663    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830665    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830672    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.830674    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.835913    4453 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.835930    4453 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.835949    4453 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.835955    4453 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836018    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836023    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836027    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836030    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836033    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836036    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836039    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836043    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836045    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836048    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.846015 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836051    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836054    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836057    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836060    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836062    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836065    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836067    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836070    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836073    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836075    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836078    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836080    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836083    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836086    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836088    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836091    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836093    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836101    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836104    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836107    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836110    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.846523 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836113    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836116    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836118    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836121    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836124    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836126    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836129    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836131    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836135    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836137    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836140    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836143    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836146    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836149    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836151    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836155    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836158    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836160    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836163    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836166    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.847038 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836169    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836171    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836174    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836177    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836180    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836183    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836186    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836188    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836191    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836193    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836196    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836198    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836201    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836204    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836206    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836209    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836212    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836214    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836217    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.847576 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836220    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836222    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836225    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836228    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836230    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836233    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836236    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836238    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836241    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836244    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836247    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836250    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836252    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836255    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836258    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836261    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836264    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836266    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836269    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836272    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.848208 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836274    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836277    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836281    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836285    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836288    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836291    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836294    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836296    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836299    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836302    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836304    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836307    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836310    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836312    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836315    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.849024 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836319    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.836361    4453 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836505    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836511    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836516    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836520    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836524    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836528    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836531    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836534    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836537    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836540    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836542    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836545    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836548    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.849575 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836551    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836554    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836556    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836559    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836562    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836565    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836567    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836570    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836573    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836575    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836578    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836580    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836583    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836586    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836589    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836591    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836595    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836599    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836602    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.849979 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836605    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836607    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836610    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836613    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836615    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836618    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836622    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836625    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836627    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836630    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836633    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836635    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836638    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836640    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836643    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836646    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836648    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836651    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836653    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836656    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836659    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.850493 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836661    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836664    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836666    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836669    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836671    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836674    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836677    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836679    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836682    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836685    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836687    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836690    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836692    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836695    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836698    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836700    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836703    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836706    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836709    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836711    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.851034 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836714    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836717    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836719    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836722    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836725    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836728    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836730    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836733    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836736    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836738    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836741    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836743    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836746    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836748    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836751    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836754    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836756    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836759    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836762    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836764    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.851791 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836767    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836770    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836772    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836775    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836777    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836780    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836783    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836785    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836788    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836790    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836793    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836796    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836799    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.836821    4453 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836853    4453 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836857    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 11:30:00.852290 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836860    4453 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836863    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836866    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836869    4453 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836871    4453 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836874    4453 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836877    4453 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836880    4453 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836882    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836885    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836888    4453 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836890    4453 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836893    4453 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836896    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836899    4453 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836901    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836904    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836907    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836909    4453 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836912    4453 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 11:30:00.852699 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836917    4453 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836919    4453 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836922    4453 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836926    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836929    4453 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836932    4453 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836934    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836939    4453 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836943    4453 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836946    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836949    4453 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836951    4453 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836954    4453 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836957    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836959    4453 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836962    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836965    4453 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836968    4453 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836971    4453 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 11:30:00.853199 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836973    4453 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836976    4453 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836979    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836981    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836984    4453 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836986    4453 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836989    4453 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836992    4453 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836994    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.836997    4453 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837000    4453 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837002    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837005    4453 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837009    4453 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837012    4453 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837016    4453 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837022    4453 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837026    4453 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837029    4453 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 11:30:00.853884 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837031    4453 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837034    4453 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837037    4453 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837040    4453 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837043    4453 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837045    4453 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837048    4453 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837051    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837053    4453 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837056    4453 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837058    4453 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837061    4453 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837064    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837066    4453 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837069    4453 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837072    4453 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837074    4453 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837077    4453 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837079    4453 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837082    4453 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837085    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 11:30:00.854702 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837087    4453 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837090    4453 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837092    4453 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837095    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837098    4453 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837102    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837104    4453 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837107    4453 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837111    4453 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837114    4453 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837117    4453 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837119    4453 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837122    4453 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837125    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837127    4453 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837131    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837133    4453 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837136    4453 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837139    4453 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837142    4453 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 11:30:00.855600 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837144    4453 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837147    4453 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837150    4453 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837153    4453 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:00.837155    4453 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.837209    4453 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838201    4453 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838255    4453 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838440    4453 server.go:1014] "Starting client certificate rotation"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838578    4453 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838618    4453 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 11:24:22 +0000 UTC" deadline="2028-04-04 13:43:10.902288928 +0000 UTC"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838655    4453 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12530h13m10.06363588s"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.838874    4453 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.840023    4453 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.841688    4453 log.go:25] "Validated CRI v1 runtime API"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.843541    4453 log.go:25] "Validated CRI v1 image API"
Oct 30 11:30:00.856226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.844217    4453 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 11:30:00.856644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.845683    4453 fs.go:135] Filesystem UUIDs: map[6350d68a-01fe-420e-86ac-068375079e1d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a293865f-bd7e-4a36-8bf2-91af6c5653fb:/dev/nvme0n1p3]
Oct 30 11:30:00.856687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.845703    4453 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/618193af9f2c4c8f8c0c2cc87026fea8ce98fb5f20409cf30eeb90d4e83b5730/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/618193af9f2c4c8f8c0c2cc87026fea8ce98fb5f20409cf30eeb90d4e83b5730/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bf75660c31abcaee9d9ecc764c0c9a164c58cf10e1bf781eb3215723f69d252/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bf75660c31abcaee9d9ecc764c0c9a164c58cf10e1bf781eb3215723f69d252/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab39c7dcc480aaecf47134b8495ef5400cd7306ab6b12e6660466721e75e84d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab39c7dcc480aaecf47134b8495ef5400cd7306ab6b12e6660466721e75e84d6/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~projected/kube-api-access-fdw9f:{mountpoint:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~projected/kube-api-access-fdw9f major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2123026e-ac5a-4ef1-80f6-76b1bcc174c1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2123026e-ac5a-4ef1-80f6-76b1bcc174c1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2970c8b3-5474-4885-95e7-9562083216e2/volumes/kubernetes.io~projected/kube-api-access-t5zrb:{mountpoint:/var/lib/kubelet/pods/2970c8b3-5474-4885-95e7-9562083216e2/volumes/kubernetes.io~projected/kube-api-access-t5zrb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~projected/kube-api-access-7ljc8:{mountpoint:/var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~projected/kube-api-access-7ljc8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f9925c-7d11-4296-9593-7a747f8d5916/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/44f9925c-7d11-4296-9593-7a747f8d5916/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5727749f-b29a-44b7-8b7e-02b65d349ed6/volumes/kubernetes.io~projected/kube-api-access-xwglp:{mountpoint:/var/lib/kubelet/pods/5727749f-b29a-44b7-8b7e-02b65d349ed6/volumes/kubernetes.io~projected/kube-api-access-xwglp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62d337ee-1eec-4cd7-943e-5046b9db2be9/volumes/kubernetes.io~projected/kube-api-access-d5bk5:{mountpoint:/var/lib/kubelet/pods/62d337ee-1eec-4cd7-943e-5046b9db2be9/volumes/kubernetes.io~projected/kube-api-access-d5bk5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cb183e6-c4c5-4b96-a2a0-12897b0572ca/volumes/kubernetes.io~projected/kube-api-access-dq46s:{mountpoint:/var/lib/kubelet/pods/7cb183e6-c4c5-4b96-a2a0-12897b0572ca/volumes/kubernetes.io~projected/kube-api-access-dq46s major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~projected/kube-api-access-bb6kn:{mountpoint:/var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~projected/kube-api-access-bb6kn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e0648c5-c6b4-49db-9dd2-4f058e911679/volumes/kubernetes.io~projected/kube-api-access-2vktr:{mountpoint:/var/lib/kubelet/pods/9e0648c5-c6b4-49db-9dd2-4f058e911679/volumes/kubernetes.io~projected/kube-api-access-2vktr major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aca0a506-f0f2-4037-b2bb-312e2532b938/volumes/kubernetes.io~projected/kube-api-access-ngvgx:{mountpoint:/var/lib/kubelet/pods/aca0a506-f0f2-4037-b2bb-312e2532b938/volumes/kubernetes.io~projected/kube-api-access-ngvgx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~projected/kube-api-access-frmfg:{mountpoint:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~projected/kube-api-access-frmfg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb76f4db-7292-45d7-8780-f5e14eedbfd3/volumes/kubernetes.io~projected/kube-api-access-cp929:{mountpoint:/var/lib/kubelet/pods/cb76f4db-7292-45d7-8780-f5e14eedbfd3/volumes/kubernetes.io~projected/kube-api-access-cp929 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~projected/kube-api-access-6g6dn:{mountpoint:/var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~projected/kube-api-access-6g6dn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~secret/signing-key major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef3a0a4d-2922-4948-af17-cf737a3f6273/volumes/kubernetes.io~projected/kube-api-access-cnzk9:{mountpoint:/var/lib/kubelet/pods/ef3a0a4d-2922-4948-af17-cf737a3f6273/volumes/kubernetes.io~projected/kube-api-access-cnzk9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7270d2a-df57-495e-82f2-0f34335e48e0/volumes/kubernetes.io~projected/kube-api-access-lkjc6:{mountpoint:/var/lib/kubelet/pods/f7270d2a-df57-495e-82f2-0f34335e48e0/volumes/kubernetes.io~projected/kube-api-access-lkjc6 major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3e943ed54ee111ca44cb23a4c31e8001aad8e375f26a3675745cb0eb8236a0b4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/43bfd2119fe2dac8079519bd4009caa86ae8e3f0bb6a1eac100351aa98e96514/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/22e54b2e579c39aeebdcf3bb4ae6782b3ad6f2e289ea48a95dcc11631cb44a0a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9df3d854082460ab1cf9242122337d1c870f37bd420f8834f42d21cd28c6c7a0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cf55926e41b3c5a5043a981581fedfbfa558e7336d1653c57660d62cd6e9c76b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0a90779702e9cbc819d1fe64de99e34be07197397a469e641cc527ee3730f605/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2c22af46f1f86801fe64492ee4e12476ffdc876d6c720a11627416d39a849408/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e50365e8577ed59df2bf9a7d30f15c76b5f6cbd50283e3527fa0762814be2abd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c0a9089dd2ec213088af5bb0989cd13b69e98d19a83845bd8742ce37c0f27c6b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0075df15bb098c71180e784ec3ad8a671505ac9cb848de0249b57eb8be7ad1f8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c928bc9c4d0e0ce1fedf1c757ab8b02dc00fb6c23406bfe964b51e096935dd42/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/454b66d058298beb8e8f21870d0c193505e184027aa920d010d99c156f7065c1/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/362a211caf33523d799323d23b8eb14fc5ffa148419d8d1bfbf1abe2c8953dd7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9fd7a1e0b72cce29ceb914b08d20752c5783bd861d72a8714940041344363c93/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/da67587ae64eebd2993a2542d09ed618a61cb6f666a717dfdd7939c431add515/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4d57460bc6c9793ff9bd8c082967bc4455b7b15a4176e7856e32a693d7946744/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8208f51cb7300dd3461a4dafab0645f5b73e53b9661c946817bdcaf9dcdd0a8e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/690eae2456957e57fd1891ea3f2b5c2ae35fd94cab483778c3206bbe686add56/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/75ea07ac0f02a6fb83621116065dc06cd4ca5a90332a3efcec283621fd3c2be7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c012e6eaf58eb2ba6d65e5dd429ebf5f8aa796194c9acf480f3cb401a4ab1ee7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c4f83cabb1896b9531ba7806e2ac4d9b7807747c392bb4267b3006ce6ceeb4a1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/6bf30e153dff09725e3d34f16c8e6521c2d9190c07a0083347878433e58881ce/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/edc6660f82093b2d83e6a571666d6a82320aa5f63e03c4527cad0a02e3b15c30/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1dedea6bff9ca1e41fc4a83b87d7957849b5bc6241bb4b2ebe2ee986ab45f9a9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/1456dda5bd840bccf7e422e5ca0587d52e00c55e0f7444535a85c8d1ba8f12ee/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/134a7a4e81182f884f10e9041e04da67379c91288acd720bb4fc20b32d699a4a/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/88d8d39ad2e9481a0460850787f7ff6cea37e7c2fa6d6e5b271a29fda0aad924/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/fe84d0b6c618067d2b5d1389465a9a64087081e1abe969110d50ae98912a2482/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/f7a9636014204e63634afa9e6de216b03f9f7ab814ba4628dafdbea736950b42/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/951fc74560fff01f82b49557ce73202f8a3008e03d5390930946be05a045c048/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5fa5b1971d313b1724cf83dd6a261b69e9c8805e1b3732c79b98b3346a8fedb5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fa7b22fc15944971fad8c4d5ba3b8743a82a1d11d442b5dc7ddd5a11987064e0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/7f5ef60fa0d48de77a1b6a51d3c4023045ae7674406fd134d3f566e9ee561bab/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/e5cdd6c7ec481a81ea9b33ad2d18169c382f78db8b0dca80bb944cab0445c095/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c38f141878f0ec59782f0af78457bf8f26c2de00079967310070405b84ee12b9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a527de840dbb0eb22da6ee137a4f392d0018ab3fdae7fcd9bde2c410f7874062/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/dd64350c691b5c76d0e84332c33308113af7cae2136ce6ccd05494b97b5eb818/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/b35a6fe1923b5bb43114cbd7dfa36cf26c83d78b3ec57d5032e8f43cc38083cb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6fe8516c02d98c2379467f4859b40e29c9bc917ebb83bb71d41bf72445481c1b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/95b4e4f342ea167e84b65a8a6a3671ecfd39a27b8321ef486b222b7756b8aeca/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853125    4453 manager.go:217] Machine: {Timestamp:2026-10-30 11:30:00.852658464 +0000 UTC m=+0.072943366 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099916 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284400a8315eaaa8cb56454fcd5930 SystemUUID:ec284400-a831-5eaa-a8cb-56454fcd5930 BootID:da411877-6f5f-46f7-b701-858aedc60db9 Filesystems:[{Device:/var/lib/kubelet/pods/44f9925c-7d11-4296-9593-7a747f8d5916/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2123026e-ac5a-4ef1-80f6-76b1bcc174c1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7270d2a-df57-495e-82f2-0f34335e48e0/volumes/kubernetes.io~projected/kube-api-access-lkjc6 DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~projected/kube-api-access-frmfg DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~projected/kube-api-access-bb6kn DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/706c57bb0db0f1d0a4571bbfffdfa531f76a804278e77211c510937ca1a45d94/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5727749f-b29a-44b7-8b7e-02b65d349ed6/volumes/kubernetes.io~projected/kube-api-access-xwglp DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4e0099a6a9c7b0f605aa93a9cb6e9e4b78d4092f7dca77b18b22ab00d107d03/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2970c8b3-5474-4885-95e7-9562083216e2/volumes/kubernetes.io~projected/kube-api-access-t5zrb DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef3a0a4d-2922-4948-af17-cf737a3f6273/volumes/kubernetes.io~projected/kube-api-access-cnzk9 DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6f8d27af7adc92938990341ab099d2de4c120d301e65fb0af195237d30f66fc/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26196f67a3b78ce3d83de47bfd1228dc73205a3362d138f509e2f19a60f4cf2e/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c2bac2667b675ac39e3edeaf6c05d66dc2ab174989be676ce298ff1f8685e74/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~projected/kube-api-access-7ljc8 DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baca0000-f0c2-4984-ac8b-558dfadf5a60/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5df0a43f7491810037c2c2c9ac94f0cf7d7fd418468e165e7d8e45b0f088676/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62d337ee-1eec-4cd7-943e-5046b9db2be9/volumes/kubernetes.io~projected/kube-api-access-d5bk5 DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38b69f25-5947-4982-8a04-bc09d51a87b3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/618193af9f2c4c8f8c0c2cc87026fea8ce98fb5f20409cf30eeb90d4e83b5730/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19508dd9e79932da9ee0a66c31da585fe740588c3518a0ed006302741acc30ea/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96e3a31c-a58f-42b7-a29d-3f41ef0d3435/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~projected/kube-api-access-fdw9f DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb76f4db-7292-45d7-8780-f5e14eedbfd3/volumes/kubernetes.io~projected/kube-api-access-cp929 DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bf75660c31abcaee9d9ecc764c0c9a164c58cf10e1bf781eb3215723f69d252/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3/volumes/kubernetes.io~projected/kube-api-access-6g6dn DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dfc650ca0f9cbb5a2d73751efc160e47155e2bc4b744de8d9a73f41223aa512/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9e0648c5-c6b4-49db-9dd2-4f058e911679/volumes/kubernetes.io~projected/kube-api-access-2vktr DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec7c97e45bbc50e44f39100c9826c97ddaa028e32944f3982a598a9dedf8d6c8/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab39c7dcc480aaecf47134b8495ef5400cd7306ab6b12e6660466721e75e84d6/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/16817ce5-34ed-46e3-9969-f4d62ea2eea9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aca0a506-f0f2-4037-b2bb-312e2532b938/volumes/kubernetes.io~projected/kube-api-access-ngvgx DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7cb183e6-c4c5-4b96-a2a0-12897b0572ca/volumes/kubernetes.io~projected/kube-api-access-dq46s DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b1479006e6509aed9a09a9ac8a1458803f2237e22bbfc0e2aa1218e1233d5f3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26196f67a3b78ce MacAddress:42:ea:1f:ca:d2:21 Speed:10000 Mtu:8901} {Name:6b1479006e6509a MacAddress:42:60:da:ce:a2:72 Speed:10000 Mtu:8901} {Name:706c57bb0db0f1d MacAddress:1e:0f:04:0b:ea:39 Speed:10000 Mtu:8901} {Name:9c2bac2667b675a MacAddress:2e:f5:ca:27:8f:98 Speed:10000 Mtu:8901} {Name:a6f8d27af7adc92 MacAddress:f2:db:9a:61:ee:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f9:12:a2:91 Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:0e:47:9e:cf:54 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f9:12:a2:91 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ba:f1:46:f1:d9:03 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:50:ba:29:f4:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853371    4453 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853413    4453 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853765    4453 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853800    4453 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.853992    4453 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854003    4453 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854018    4453 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854044    4453 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854168    4453 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854411    4453 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854557    4453 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854602    4453 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854616    4453 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854630    4453 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854643    4453 kubelet.go:405] "Adding apiserver pod source"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.854654    4453 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.855484    4453 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.855497    4453 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:30:00.856937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.856049    4453 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 11:30:00.865563 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865546    4453 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 11:30:00.865563 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865560    4453 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 11:30:00.865679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865655    4453 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 11:30:00.865850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865839    4453 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 11:30:00.865887 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865868    4453 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 11:30:00.865973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865962    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 11:30:00.866001 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865983    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 11:30:00.866001 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865992    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 11:30:00.866001 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.865999    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866009    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866016    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866024    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866032    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866040    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866046    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866054    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 11:30:00.866080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866064    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 11:30:00.866424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866089    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 11:30:00.866424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.866095    4453 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 11:30:00.869490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.869477    4453 server.go:1263] "Started kubelet"
Oct 30 11:30:00.869729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.869620    4453 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 11:30:00.869836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.869751    4453 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 11:30:00.870105 ip-10-0-143-18 systemd[1]: Started Kubernetes Kubelet.
Oct 30 11:30:00.870453 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.870306    4453 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 11:30:00.870453 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.870415    4453 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 11:30:00.873569 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.873551    4453 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 11:30:00.874160 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.873887    4453 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 11:30:00.874446 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:00.874345    4453 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-18.ec2.internal\" not found"
Oct 30 11:30:00.874446 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.874419    4453 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 11:30:00.874446 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.874428    4453 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 11:30:00.875354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.874651    4453 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 11:30:00.876578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.876556    4453 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 11:30:00.876871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.876821    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb183e6-c4c5-4b96-a2a0-12897b0572ca" volumeName="kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877062    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16817ce5-34ed-46e3-9969-f4d62ea2eea9" volumeName="kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877166    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e0648c5-c6b4-49db-9dd2-4f058e911679" volumeName="kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877191    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baca0000-f0c2-4984-ac8b-558dfadf5a60" volumeName="kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877213    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3" volumeName="kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877263    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7270d2a-df57-495e-82f2-0f34335e48e0" volumeName="kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877282    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16817ce5-34ed-46e3-9969-f4d62ea2eea9" volumeName="kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877298    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f9925c-7d11-4296-9593-7a747f8d5916" volumeName="kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877374    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baca0000-f0c2-4984-ac8b-558dfadf5a60" volumeName="kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877417    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76f4db-7292-45d7-8780-f5e14eedbfd3" volumeName="kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877432    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16817ce5-34ed-46e3-9969-f4d62ea2eea9" volumeName="kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877463    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e3a31c-a58f-42b7-a29d-3f41ef0d3435" volumeName="kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877477    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a506-f0f2-4037-b2bb-312e2532b938" volumeName="kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877521    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baca0000-f0c2-4984-ac8b-558dfadf5a60" volumeName="kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877559    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef3a0a4d-2922-4948-af17-cf737a3f6273" volumeName="kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877583    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b69f25-5947-4982-8a04-bc09d51a87b3" volumeName="kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877599    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b69f25-5947-4982-8a04-bc09d51a87b3" volumeName="kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877655    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5727749f-b29a-44b7-8b7e-02b65d349ed6" volumeName="kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config" seLinuxMountContext=""
Oct 30 11:30:00.877711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877679    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d337ee-1eec-4cd7-943e-5046b9db2be9" volumeName="kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877724    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cb183e6-c4c5-4b96-a2a0-12897b0572ca" volumeName="kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877760    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a506-f0f2-4037-b2bb-312e2532b938" volumeName="kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877790    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a506-f0f2-4037-b2bb-312e2532b938" volumeName="kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877804    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3" volumeName="kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877842    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16817ce5-34ed-46e3-9969-f4d62ea2eea9" volumeName="kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877860    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b69f25-5947-4982-8a04-bc09d51a87b3" volumeName="kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877967    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5727749f-b29a-44b7-8b7e-02b65d349ed6" volumeName="kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878024    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d337ee-1eec-4cd7-943e-5046b9db2be9" volumeName="kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878038    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62d337ee-1eec-4cd7-943e-5046b9db2be9" volumeName="kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878081    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96e3a31c-a58f-42b7-a29d-3f41ef0d3435" volumeName="kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878114    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aca0a506-f0f2-4037-b2bb-312e2532b938" volumeName="kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878128    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3" volumeName="kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878152    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16817ce5-34ed-46e3-9969-f4d62ea2eea9" volumeName="kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878181    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f9925c-7d11-4296-9593-7a747f8d5916" volumeName="kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878220    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5727749f-b29a-44b7-8b7e-02b65d349ed6" volumeName="kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878242    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2123026e-ac5a-4ef1-80f6-76b1bcc174c1" volumeName="kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878254    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2970c8b3-5474-4885-95e7-9562083216e2" volumeName="kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878282    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38b69f25-5947-4982-8a04-bc09d51a87b3" volumeName="kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878358    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb76f4db-7292-45d7-8780-f5e14eedbfd3" volumeName="kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878400    4453 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef3a0a4d-2922-4948-af17-cf737a3f6273" volumeName="kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9" seLinuxMountContext=""
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878413    4453 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 11:30:00.878551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878421    4453 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.877465    4453 factory.go:55] Registering systemd factory
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.878631    4453 factory.go:223] Registration of the systemd container factory successfully
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879015    4453 factory.go:153] Registering CRI-O factory
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879030    4453 factory.go:223] Registration of the crio container factory successfully
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879114    4453 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879140    4453 factory.go:103] Registering Raw factory
Oct 30 11:30:00.879537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879160    4453 manager.go:1199] Started watching for new ooms in manager
Oct 30 11:30:00.879725 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.879572    4453 manager.go:319] Starting recovery of all containers
Oct 30 11:30:00.880931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.880883    4453 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 11:24:23 +0000 UTC" deadline="2028-04-28 03:47:13.205602262 +0000 UTC"
Oct 30 11:30:00.880931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.880930    4453 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13096h17m12.32467592s"
Oct 30 11:30:00.901949 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.901909    4453 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 11:30:00.903526 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.903500    4453 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 11:30:00.903526 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.903528    4453 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 11:30:00.903731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.903554    4453 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 11:30:00.903731 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:00.903601    4453 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 11:30:00.917099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.917080    4453 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:30:00.917226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.917157    4453 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 11:30:00.918310 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.918290    4453 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:30:00.920490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.920462    4453 generic.go:356] "Generic (PLEG): container finished" podID="78cd7a0c462fc7ca3f08389f12997e13" containerID="4cc436a8634c85d634153fbd69eb8ae3cd68aa2b0daba46219214d7cb5457b62" exitCode=0
Oct 30 11:30:00.928125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928098    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="5471475adefd53bf85dac5f2e2d95e7a5c2c8ac9b5687ac4ba19a61082ece593" exitCode=0
Oct 30 11:30:00.928225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928126    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="4f9d5cd1edc27e181a50f2ea6dd0da93ab0703caa862a23af43e6340199bcfc7" exitCode=0
Oct 30 11:30:00.928225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928137    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="45aa41903563e332519e982dd552d864a3ad85dcfcfb08b134fdcafb84c0a1a2" exitCode=0
Oct 30 11:30:00.928225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928148    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="e381d8eba4171a63c3be25c98828f19e038ccdaa578e05d04132bc9283afdda2" exitCode=0
Oct 30 11:30:00.928225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928156    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="4063de210c57fb2d8b118049d43462b3ab59428cbbe4d64cd734291a1d744d0c" exitCode=0
Oct 30 11:30:00.928225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.928164    4453 generic.go:356] "Generic (PLEG): container finished" podID="aca0a506-f0f2-4037-b2bb-312e2532b938" containerID="508e57d0ca13b9f5c261c6638ca65b89d01378bfc96b3f4b4898df0e86d20eea" exitCode=0
Oct 30 11:30:00.930404 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.930384    4453 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:30:00.930525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.930483    4453 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 11:30:00.948223 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.948207    4453 manager.go:324] Recovery completed
Oct 30 11:30:00.961531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961510    4453 cpu_manager.go:226] "Starting" policy="none"
Oct 30 11:30:00.961531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961523    4453 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 11:30:00.961665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961544    4453 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 11:30:00.961665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961688    4453 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961704    4453 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961728    4453 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961735    4453 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961744    4453 policy_none.go:50] "Start"
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961762    4453 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 11:30:00.961837 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.961776    4453 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 11:30:00.966822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.966808    4453 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 11:30:00.966892 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.966826    4453 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 11:30:00.966892 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.966835    4453 policy_none.go:44] "Start"
Oct 30 11:30:00.971131 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.970706    4453 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 11:30:00.976354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976296    4453 manager.go:341] "Starting Device Plugin manager"
Oct 30 11:30:00.976354 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:00.976319    4453 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 11:30:00.976354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976343    4453 server.go:87] "Starting device plugin registration server"
Oct 30 11:30:00.976609 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976597    4453 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 11:30:00.976688 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976608    4453 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 11:30:00.976745 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976722    4453 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 11:30:00.976828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976793    4453 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 11:30:00.976828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976826    4453 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 11:30:00.976828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976832    4453 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 11:30:00.977012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.976867    4453 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T11:30:00.976818183Z","UUID":"fdb43c7a-f594-4d6b-905a-55ad11163df1","Handler":null,"Name":"","Endpoint":""}
Oct 30 11:30:00.978037 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:00.978019    4453 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 11:30:00.980532 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.980510    4453 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 11:30:00.980532 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:00.980539    4453 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 11:30:01.004161 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004119    4453 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"]
Oct 30 11:30:01.004700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004667    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerStarted","Data":"3c1fd474751bd7561c5108f608ee26b07e8e911dd98f7d73c2531d6daa168b2c"}
Oct 30 11:30:01.004746 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004713    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerDied","Data":"4cc436a8634c85d634153fbd69eb8ae3cd68aa2b0daba46219214d7cb5457b62"}
Oct 30 11:30:01.004746 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004738    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal" event={"ID":"78cd7a0c462fc7ca3f08389f12997e13","Type":"ContainerStarted","Data":"406296547ad2d6761456358a9a140e2e99f545b1d9c1f41621af3f2d7b3cc765"}
Oct 30 11:30:01.004809 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004790    4453 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.004838 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004815    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" event={"ID":"6905fd60ff57bade2ab9f0792ffa70c2","Type":"ContainerStarted","Data":"459e07ecd005cfa3bc2be292bb8495d0624229eb860944d14cec48dcb0ff2080"}
Oct 30 11:30:01.004865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004836    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" event={"ID":"6905fd60ff57bade2ab9f0792ffa70c2","Type":"ContainerStarted","Data":"5255499c7cf34af2c42f5f96875a82817bef9f9412e10facd2d057c2d634a53b"}
Oct 30 11:30:01.004893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.004879    4453 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.027899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.027878    4453 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:30:01.027987 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:01.027918    4453 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.028898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.028884    4453 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:30:01.028941 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:01.028920    4453 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.077672 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.077648    4453 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 11:30:01.078894 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.078877    4453 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 11:30:01.078988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.078906    4453 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 11:30:01.078988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.078916    4453 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-18.ec2.internal" event="NodeHasSufficientPID"
Oct 30 11:30:01.078988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.078951    4453 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.115258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.115238    4453 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.115360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.115290    4453 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.118946 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.118925    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.119051 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.118950    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.119051 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.118968    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.219960 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.219924    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.219960 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.219965    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.220189 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.219986    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.220189 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.220032    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6905fd60ff57bade2ab9f0792ffa70c2-config\") pod \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" (UID: \"6905fd60ff57bade2ab9f0792ffa70c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.220189 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.220046    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.220189 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.220068    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78cd7a0c462fc7ca3f08389f12997e13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" (UID: \"78cd7a0c462fc7ca3f08389f12997e13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.855400 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.855362    4453 apiserver.go:52] "Watching apiserver"
Oct 30 11:30:01.895414 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.895386    4453 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 11:30:01.895814 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.895797    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d5gjj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal","openshift-image-registry/node-ca-8bfdd","openshift-service-ca/service-ca-5fffc97bdf-4fjgw","openshift-cluster-node-tuning-operator/tuned-bkdnv","openshift-network-diagnostics/network-check-target-qq4wf","openshift-ovn-kubernetes/ovnkube-node-9gwhb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d","kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal","openshift-insights/insights-runtime-extractor-fv5lx","openshift-multus/multus-l5l6p","openshift-network-operator/iptables-alerter-ptd94","kube-system/global-pull-secret-syncer-hgkvl","openshift-dns/node-resolver-h6llc","openshift-ingress-canary/ingress-canary-lvpqd","openshift-multus/network-metrics-daemon-wfrcp","openshift-multus/multus-additional-cni-plugins-d4wr8","kube-system/konnectivity-agent-6d7hf"]
Oct 30 11:30:01.901944 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.901916    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:01.903665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.903644    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:01.910673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.910655    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 11:30:01.910798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.910781    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 11:30:01.910957 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.910945    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.911180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.911166    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.911228 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.911200    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-899rd\""
Oct 30 11:30:01.911273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.911260    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 11:30:01.911629 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.911615    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-k7ct5\""
Oct 30 11:30:01.911676 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.911660    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.923934 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.923917    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.924031 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.923960    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 11:30:01.924031 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924029    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.924230 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924213    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bltqf\""
Oct 30 11:30:01.924278 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924217    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hc7n5\""
Oct 30 11:30:01.924322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924276    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 11:30:01.924322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924281    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 11:30:01.924441 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924322    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 11:30:01.924517 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924492    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 11:30:01.924517 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924501    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 11:30:01.924656 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.924644    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 11:30:01.925082 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925070    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 11:30:01.925127 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925090    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.925185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925172    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 11:30:01.925272 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925261    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 11:30:01.925317 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925296    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.925582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925567    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 11:30:01.925780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925762    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hnpqt\""
Oct 30 11:30:01.925899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925821    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dtb5\""
Oct 30 11:30:01.925899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925884    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.926002 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925953    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 11:30:01.926002 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.925991    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wql9s\""
Oct 30 11:30:01.926085 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926077    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926135 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926081    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xq7t5\""
Oct 30 11:30:01.926135 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926120    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.926246 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926228    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.926308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926233    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 11:30:01.926308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926289    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.926308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926299    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926311    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qxb8z\""
Oct 30 11:30:01.926454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926326    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926344    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 11:30:01.926454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926430    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926446    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lmhtb\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926466    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926476    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926472    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926475    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926501    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.926652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.926501    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6tzdw\""
Oct 30 11:30:01.936025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936008    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 11:30:01.936025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936022    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 11:30:01.936116 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936032    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hsk2h\""
Oct 30 11:30:01.936402 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936387    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4s6wd\""
Oct 30 11:30:01.936402 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936395    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936389    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkn76\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936411    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936428    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936391    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936439    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936463    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlvg9\""
Oct 30 11:30:01.936518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.936500    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hm9cg\""
Oct 30 11:30:01.941664 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.941643    4453 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.941782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.941771    4453 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.964631 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.964611    4453 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:30:01.964716 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:01.964654    4453 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-18.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.964716 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.964617    4453 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 11:30:01.964782 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:01.964722    4453 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-18.ec2.internal"
Oct 30 11:30:01.976629 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:01.976614    4453 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 11:30:02.023115 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023090    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq46s\" (UniqueName: \"kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.023248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023121    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.023248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023142    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.023248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023170    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.023248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023189    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zrb\" (UniqueName: \"kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.023248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023204    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023251    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023280    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-tmp\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023360    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023402    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023427    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:30:02.023514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023474    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023525    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-key\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023525    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023583    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023620    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023652    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023682    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.023747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023738    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023761    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023777    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023795    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023811    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023828    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023862    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023926    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb76f4db-7292-45d7-8780-f5e14eedbfd3-serviceca\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.023936    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw9f\" (UniqueName: \"kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024011    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024028    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024070    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024099    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef3a0a4d-2922-4948-af17-cf737a3f6273-tmp-dir\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024127    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024138    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024158    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024185    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-daemon-config\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024252    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024297    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024367    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5727749f-b29a-44b7-8b7e-02b65d349ed6-cni-binary-copy\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024372    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.024487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024457    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp929\" (UniqueName: \"kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024514    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024543    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024565    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024587    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-script-lib\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024593    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024618    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024671    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38b69f25-5947-4982-8a04-bc09d51a87b3-tmp-dir\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024673    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024710    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024747    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bk5\" (UniqueName: \"kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024770    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g6dn\" (UniqueName: \"kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024785    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/44f9925c-7d11-4296-9593-7a747f8d5916-agent-certs\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024800    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024825    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38b69f25-5947-4982-8a04-bc09d51a87b3-config-volume\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024894    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024946    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024975    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.024989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.024949    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-env-overrides\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025009    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6kn\" (UniqueName: \"kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025026    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025065    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025104    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025162    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025186    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025204    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-original-pull-secret\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025212    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025231    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovnkube-config\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025237    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025274    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025320    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025373    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025398    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025424    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025456    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwglp\" (UniqueName: \"kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025474    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025492    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025521    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025558    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025615    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzk9\" (UniqueName: \"kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.025662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025672    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025691    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frmfg\" (UniqueName: \"kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025706    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025723    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvgx\" (UniqueName: \"kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025726    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16817ce5-34ed-46e3-9969-f4d62ea2eea9-ovn-node-metrics-cert\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025841    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025876    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025902    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025924    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljc8\" (UniqueName: \"kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025947    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025964    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025979    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.025978    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-iptables-alerter-script\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026036    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026068    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026100    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026125    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026152    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026178    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026205    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026214    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-cert\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026229    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/44f9925c-7d11-4296-9593-7a747f8d5916-konnectivity-ca\") pod \"konnectivity-agent-6d7hf\" (UID: \"44f9925c-7d11-4296-9593-7a747f8d5916\") " pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026245    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjc6\" (UniqueName: \"kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.026659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026272    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026691    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aca0a506-f0f2-4037-b2bb-312e2532b938-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026691    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38b69f25-5947-4982-8a04-bc09d51a87b3-metrics-tls\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026728    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026806    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026851    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026884    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026958    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.026995    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027054    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027107    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027160    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027197    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027209    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-signing-cabundle\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027245    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027324    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027552    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/62d337ee-1eec-4cd7-943e-5046b9db2be9-data-volume\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.027660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.027617    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-tuned\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.048771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.048739    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljc8\" (UniqueName: \"kubernetes.io/projected/38b69f25-5947-4982-8a04-bc09d51a87b3-kube-api-access-7ljc8\") pod \"dns-default-d5gjj\" (UID: \"38b69f25-5947-4982-8a04-bc09d51a87b3\") " pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:02.048885 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.048808    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq46s\" (UniqueName: \"kubernetes.io/projected/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-kube-api-access-dq46s\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.048982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.048957    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw9f\" (UniqueName: \"kubernetes.io/projected/16817ce5-34ed-46e3-9969-f4d62ea2eea9-kube-api-access-fdw9f\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.049045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.049029    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmfg\" (UniqueName: \"kubernetes.io/projected/baca0000-f0c2-4984-ac8b-558dfadf5a60-kube-api-access-frmfg\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.049282 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.049263    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g6dn\" (UniqueName: \"kubernetes.io/projected/cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3-kube-api-access-6g6dn\") pod \"service-ca-5fffc97bdf-4fjgw\" (UID: \"cdd6fb8f-85c3-4a50-80b7-fca2fdc580a3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4fjgw"
Oct 30 11:30:02.050734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.050715    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zrb\" (UniqueName: \"kubernetes.io/projected/2970c8b3-5474-4885-95e7-9562083216e2-kube-api-access-t5zrb\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.051200 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.051183    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vktr\" (UniqueName: \"kubernetes.io/projected/9e0648c5-c6b4-49db-9dd2-4f058e911679-kube-api-access-2vktr\") pod \"network-check-target-qq4wf\" (UID: \"9e0648c5-c6b4-49db-9dd2-4f058e911679\") " pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:30:02.051386 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.051367    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6kn\" (UniqueName: \"kubernetes.io/projected/96e3a31c-a58f-42b7-a29d-3f41ef0d3435-kube-api-access-bb6kn\") pod \"ingress-canary-lvpqd\" (UID: \"96e3a31c-a58f-42b7-a29d-3f41ef0d3435\") " pod="openshift-ingress-canary/ingress-canary-lvpqd"
Oct 30 11:30:02.054025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054005    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwglp\" (UniqueName: \"kubernetes.io/projected/5727749f-b29a-44b7-8b7e-02b65d349ed6-kube-api-access-xwglp\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.054140 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054062    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjc6\" (UniqueName: \"kubernetes.io/projected/f7270d2a-df57-495e-82f2-0f34335e48e0-kube-api-access-lkjc6\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.054140 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054111    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp929\" (UniqueName: \"kubernetes.io/projected/cb76f4db-7292-45d7-8780-f5e14eedbfd3-kube-api-access-cp929\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.054401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054378    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzk9\" (UniqueName: \"kubernetes.io/projected/ef3a0a4d-2922-4948-af17-cf737a3f6273-kube-api-access-cnzk9\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.054575 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054555    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvgx\" (UniqueName: \"kubernetes.io/projected/aca0a506-f0f2-4037-b2bb-312e2532b938-kube-api-access-ngvgx\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.054637 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.054614    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bk5\" (UniqueName: \"kubernetes.io/projected/62d337ee-1eec-4cd7-943e-5046b9db2be9-kube-api-access-d5bk5\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.127971 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.127943    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.127971 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.127974    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.127993    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128007    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128071    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-kubelet\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128115    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-multus-certs\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128145    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-cnibin\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128161    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128179    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128195    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128203    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-systemd-units\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128214    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128245    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-registration-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128255    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysconfig\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128260    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128286    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128317    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-systemd\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128314    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128287    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128325    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128364    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128373    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-hostroot\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128389    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128365    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128392    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-netd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128409    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128436    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128448    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-netns\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128474    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128508    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:02.128514    4453 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:02.128528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128531    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128552    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128569    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128594    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128571    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-systemd\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128516    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef3a0a4d-2922-4948-af17-cf737a3f6273-hosts-file\") pod \"node-resolver-h6llc\" (UID: \"ef3a0a4d-2922-4948-af17-cf737a3f6273\") " pod="openshift-dns/node-resolver-h6llc"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128594    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128618    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-kubelet\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128632    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-node-log\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128645    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-sysctl-conf\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128553    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-run-netns\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:02.128664    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:02.628626404 +0000 UTC m=+1.848911292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128685    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128699    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-etc-kubernetes\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128709    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128737    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128758    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-var-lib-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128760    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128774    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-multus\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128802    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128805    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-socket-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128845    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128845    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128865    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128885    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb76f4db-7292-45d7-8780-f5e14eedbfd3-host\") pod \"node-ca-8bfdd\" (UID: \"cb76f4db-7292-45d7-8780-f5e14eedbfd3\") " pod="openshift-image-registry/node-ca-8bfdd"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128889    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128890    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-system-cni-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128914    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128920    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-kubernetes\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128931    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128946    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128961    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.128987    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-run\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129011    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129022    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-system-cni-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129034    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129042    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-sys-fs\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129036    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-dbus\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129054    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129072    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129088    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-slash\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129095    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-etc-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129089    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129094    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2123026e-ac5a-4ef1-80f6-76b1bcc174c1-kubelet-config\") pod \"global-pull-secret-syncer-hgkvl\" (UID: \"2123026e-ac5a-4ef1-80f6-76b1bcc174c1\") " pod="kube-system/global-pull-secret-syncer-hgkvl"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129127    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129133    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-cnibin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129158    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129172    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-socket-dir-parent\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129181    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129198    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-var-lib-cni-bin\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129202    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-lib-modules\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129210    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129231    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129240    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aca0a506-f0f2-4037-b2bb-312e2532b938-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4wr8\" (UID: \"aca0a506-f0f2-4037-b2bb-312e2532b938\") " pod="openshift-multus/multus-additional-cni-plugins-d4wr8"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129247    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129267    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-log-socket\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129272    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129315    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129287    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129355    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129288    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7cb183e6-c4c5-4b96-a2a0-12897b0572ca-host-slash\") pod \"iptables-alerter-ptd94\" (UID: \"7cb183e6-c4c5-4b96-a2a0-12897b0572ca\") " pod="openshift-network-operator/iptables-alerter-ptd94"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129390    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129404    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-host\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129416    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129391    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-var-lib-kubelet\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129437    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129441    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-multus-conf-dir\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129459    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.129407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129465    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2970c8b3-5474-4885-95e7-9562083216e2-device-dir\") pod \"aws-ebs-csi-driver-node-ncc7d\" (UID: \"2970c8b3-5474-4885-95e7-9562083216e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ncc7d"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129482    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-sys\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129482    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129494    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5727749f-b29a-44b7-8b7e-02b65d349ed6-host-run-k8s-cni-cncf-io\") pod \"multus-l5l6p\" (UID: \"5727749f-b29a-44b7-8b7e-02b65d349ed6\") " pod="openshift-multus/multus-l5l6p"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129512    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129514    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-openvswitch\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129548    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-run-ovn\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129575    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129600    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129621    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129625    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16817ce5-34ed-46e3-9969-f4d62ea2eea9-host-cni-bin\") pod \"ovnkube-node-9gwhb\" (UID: \"16817ce5-34ed-46e3-9969-f4d62ea2eea9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129638    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/62d337ee-1eec-4cd7-943e-5046b9db2be9-crio-socket\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129630    4453 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 11:30:02.131381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.129773    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/baca0000-f0c2-4984-ac8b-558dfadf5a60-etc-modprobe-d\") pod \"tuned-bkdnv\" (UID: \"baca0000-f0c2-4984-ac8b-558dfadf5a60\") " pod="openshift-cluster-node-tuning-operator/tuned-bkdnv"
Oct 30 11:30:02.131879 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.131861    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7270d2a-df57-495e-82f2-0f34335e48e0-metrics-certs\") pod \"network-metrics-daemon-wfrcp\" (UID: \"f7270d2a-df57-495e-82f2-0f34335e48e0\") " pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.202388 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.202357    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wfrcp"
Oct 30 11:30:02.205135 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.205115    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:30:02.558622 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.558594    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wfrcp"]
Oct 30 11:30:02.559680 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:02.559653    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7270d2a_df57_495e_82f2_0f34335e48e0.slice/crio-fe3dfd53b33bdd81f66233fa9e920563172783daffe28b9c9a9f22ea0096dc0b WatchSource:0}: Error finding container fe3dfd53b33bdd81f66233fa9e920563172783daffe28b9c9a9f22ea0096dc0b: Status 404 returned error can't find the container with id fe3dfd53b33bdd81f66233fa9e920563172783daffe28b9c9a9f22ea0096dc0b
Oct 30 11:30:02.634474 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.634446    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:02.634623 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:02.634594    4453 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:02.634671 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:02.634658    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:03.634642834 +0000 UTC m=+2.854927722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:02.945613 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:02.945046    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfrcp" event={"ID":"f7270d2a-df57-495e-82f2-0f34335e48e0","Type":"ContainerStarted","Data":"fe3dfd53b33bdd81f66233fa9e920563172783daffe28b9c9a9f22ea0096dc0b"}
Oct 30 11:30:03.353623 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:03.353591    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:30:03.642533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:03.642491    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:03.642697 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:03.642647    4453 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:03.642756 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:03.642747    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:05.642728271 +0000 UTC m=+4.863013181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:04.951958 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:04.951922    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfrcp" event={"ID":"f7270d2a-df57-495e-82f2-0f34335e48e0","Type":"ContainerStarted","Data":"1f8617317b093a3227ccb775ece0cf57d739ac79165e2acdecc3c6034381900a"}
Oct 30 11:30:04.952385 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:04.951986    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wfrcp" event={"ID":"f7270d2a-df57-495e-82f2-0f34335e48e0","Type":"ContainerStarted","Data":"563ce246297397b2980d1977c2857fe6a185fa16a32fcf4780806486e9423468"}
Oct 30 11:30:04.953617 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:04.953593    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d5gjj" event={"ID":"38b69f25-5947-4982-8a04-bc09d51a87b3","Type":"ContainerStarted","Data":"86913f08df80315fa073e492b84691621fbf24a15a2fdaaae891196c478ae3be"}
Oct 30 11:30:04.953720 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:04.953641    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d5gjj" event={"ID":"38b69f25-5947-4982-8a04-bc09d51a87b3","Type":"ContainerStarted","Data":"c1321f03aed99ce78133832ea7ff29cd9efd1b074f45fe7d72d1a9fdaa0dbdae"}
Oct 30 11:30:04.953813 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:04.953798    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:05.659247 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:05.659210    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:05.659472 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:05.659315    4453 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:05.659472 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:05.659389    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls podName:62d337ee-1eec-4cd7-943e-5046b9db2be9 nodeName:}" failed. No retries permitted until 2026-10-30 11:30:09.659375559 +0000 UTC m=+8.879660461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fv5lx" (UID: "62d337ee-1eec-4cd7-943e-5046b9db2be9") : secret "insights-runtime-extractor-tls" not found
Oct 30 11:30:09.358406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:09.358377    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:09.682625 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:09.682517    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:09.684891 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:09.684865    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/62d337ee-1eec-4cd7-943e-5046b9db2be9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv5lx\" (UID: \"62d337ee-1eec-4cd7-943e-5046b9db2be9\") " pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:09.704835 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:09.704812    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fv5lx"
Oct 30 11:30:09.997875 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:09.997849    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fv5lx"]
Oct 30 11:30:09.999997 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:09.999963    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d337ee_1eec_4cd7_943e_5046b9db2be9.slice/crio-f919f900dae1b22d2a5749df18c479771b54436e6f84cffc0d75710c023e8489 WatchSource:0}: Error finding container f919f900dae1b22d2a5749df18c479771b54436e6f84cffc0d75710c023e8489: Status 404 returned error can't find the container with id f919f900dae1b22d2a5749df18c479771b54436e6f84cffc0d75710c023e8489
Oct 30 11:30:10.971586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:10.971491    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv5lx" event={"ID":"62d337ee-1eec-4cd7-943e-5046b9db2be9","Type":"ContainerStarted","Data":"90b5eef1be977277dd7a6aa25c65fd78f6773584b5c9b8dc1c3c198f25e0c4d6"}
Oct 30 11:30:10.971586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:10.971584    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv5lx" event={"ID":"62d337ee-1eec-4cd7-943e-5046b9db2be9","Type":"ContainerStarted","Data":"7b67530ae2e7c051f28557bb5a110bbdd03b9941d041db9630decd6e60bcbe7f"}
Oct 30 11:30:10.971982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:10.971603    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv5lx" event={"ID":"62d337ee-1eec-4cd7-943e-5046b9db2be9","Type":"ContainerStarted","Data":"f919f900dae1b22d2a5749df18c479771b54436e6f84cffc0d75710c023e8489"}
Oct 30 11:30:11.023274 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:11.023247    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:11.023491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:11.023476    4453 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 11:30:11.976664 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:11.976636    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv5lx" event={"ID":"62d337ee-1eec-4cd7-943e-5046b9db2be9","Type":"ContainerStarted","Data":"9ca2eec91992a3258383f1b26d6452d7b0f39c3bf679a78d9c1c60239acf57c3"}
Oct 30 11:30:12.676071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:12.676036    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6d7hf"
Oct 30 11:30:13.621637 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.621609    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-kpjxg"]
Oct 30 11:30:13.624553 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.624533    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.633687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.633664    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"]
Oct 30 11:30:13.636498 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.636473    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:13.648175 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.648155    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-ngx7l\""
Oct 30 11:30:13.652299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.652279    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 11:30:13.660477 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.660458    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 11:30:13.660898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.660882    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kpjxg"]
Oct 30 11:30:13.662833 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.662813    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"]
Oct 30 11:30:13.709812 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.709788    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e534504-8704-4af2-8d4c-a2bd32767adb-tmp\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.709965 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.709858    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7dh\" (UniqueName: \"kubernetes.io/projected/5e534504-8704-4af2-8d4c-a2bd32767adb-kube-api-access-2x7dh\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.732927 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.732897    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58b589f7fb-84kq8"]
Oct 30 11:30:13.735729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.735712    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.761256 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.761236    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 11:30:13.761368 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.761303    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6mjtq\""
Oct 30 11:30:13.810550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.810527    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e534504-8704-4af2-8d4c-a2bd32767adb-tmp\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.810663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.810564    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/232e3b46-20e1-450a-9d33-13d6c271ea30-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vhbth\" (UID: \"232e3b46-20e1-450a-9d33-13d6c271ea30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:13.810663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.810613    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7dh\" (UniqueName: \"kubernetes.io/projected/5e534504-8704-4af2-8d4c-a2bd32767adb-kube-api-access-2x7dh\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.811033 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.811013    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e534504-8704-4af2-8d4c-a2bd32767adb-tmp\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.813768 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.813747    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 11:30:13.814229 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.814212    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b589f7fb-84kq8"]
Oct 30 11:30:13.845246 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.845218    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7dh\" (UniqueName: \"kubernetes.io/projected/5e534504-8704-4af2-8d4c-a2bd32767adb-kube-api-access-2x7dh\") pod \"downloads-6dd789c8b7-kpjxg\" (UID: \"5e534504-8704-4af2-8d4c-a2bd32767adb\") " pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.848209 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.848190    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 11:30:13.848489 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.848470    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-chtrx\""
Oct 30 11:30:13.848549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.848494    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 11:30:13.848549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.848520    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 11:30:13.911899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911838    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-image-registry-private-configuration\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.911899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911866    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-registry-certificates\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.911899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911886    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-registry-tls\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.911899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911900    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-bound-sa-token\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.912122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911922    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-installation-pull-secrets\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.912122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.911982    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/232e3b46-20e1-450a-9d33-13d6c271ea30-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vhbth\" (UID: \"232e3b46-20e1-450a-9d33-13d6c271ea30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:13.912122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.912011    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48ndn\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-kube-api-access-48ndn\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.912122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.912046    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c34812b-7d7c-4743-bcea-78110c211de4-ca-trust-extracted\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.912122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.912062    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-trusted-ca\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:13.914312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.914292    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/232e3b46-20e1-450a-9d33-13d6c271ea30-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vhbth\" (UID: \"232e3b46-20e1-450a-9d33-13d6c271ea30\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:13.935722 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.935697    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:13.946497 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:13.946477    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:14.013269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013230    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48ndn\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-kube-api-access-48ndn\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013390 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013312    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c34812b-7d7c-4743-bcea-78110c211de4-ca-trust-extracted\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013390 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013359    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-trusted-ca\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013458 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013428    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-image-registry-private-configuration\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013494 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013456    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-registry-certificates\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013494 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013486    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-registry-tls\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013511    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-bound-sa-token\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.013566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.013540    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-installation-pull-secrets\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.016574 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.016140    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-trusted-ca\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.016998 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.016934    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c34812b-7d7c-4743-bcea-78110c211de4-registry-certificates\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.017394 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.017318    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c34812b-7d7c-4743-bcea-78110c211de4-ca-trust-extracted\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.018351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.018194    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-image-registry-private-configuration\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.018619 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.018598    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c34812b-7d7c-4743-bcea-78110c211de4-installation-pull-secrets\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.018961 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.018944    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-registry-tls\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.044881 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.044855    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-bound-sa-token\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.045199 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.045175    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48ndn\" (UniqueName: \"kubernetes.io/projected/9c34812b-7d7c-4743-bcea-78110c211de4-kube-api-access-48ndn\") pod \"image-registry-58b589f7fb-84kq8\" (UID: \"9c34812b-7d7c-4743-bcea-78110c211de4\") " pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.046937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.046920    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:14.075891 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.075871    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9gwhb"
Oct 30 11:30:14.244593 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.244565    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-kpjxg"]
Oct 30 11:30:14.245430 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:14.245402    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e534504_8704_4af2_8d4c_a2bd32767adb.slice/crio-a1c7bc5a5b19e9915e6222019fef4b4da1eaaf99f67303611cfe4a2309abe6ba WatchSource:0}: Error finding container a1c7bc5a5b19e9915e6222019fef4b4da1eaaf99f67303611cfe4a2309abe6ba: Status 404 returned error can't find the container with id a1c7bc5a5b19e9915e6222019fef4b4da1eaaf99f67303611cfe4a2309abe6ba
Oct 30 11:30:14.259671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.259644    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"]
Oct 30 11:30:14.260157 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:14.260134    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232e3b46_20e1_450a_9d33_13d6c271ea30.slice/crio-0509516c0449c2fd1af45a74d1b2c2af7edacca8ef9eea7b37011a434c9ffd4b WatchSource:0}: Error finding container 0509516c0449c2fd1af45a74d1b2c2af7edacca8ef9eea7b37011a434c9ffd4b: Status 404 returned error can't find the container with id 0509516c0449c2fd1af45a74d1b2c2af7edacca8ef9eea7b37011a434c9ffd4b
Oct 30 11:30:14.345427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.345395    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b589f7fb-84kq8"]
Oct 30 11:30:14.346011 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:14.345983    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c34812b_7d7c_4743_bcea_78110c211de4.slice/crio-c9928f1b7c5e6133df748738e2511ec6d45a7e2b78f374ff062662ceba25c357 WatchSource:0}: Error finding container c9928f1b7c5e6133df748738e2511ec6d45a7e2b78f374ff062662ceba25c357: Status 404 returned error can't find the container with id c9928f1b7c5e6133df748738e2511ec6d45a7e2b78f374ff062662ceba25c357
Oct 30 11:30:14.960389 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.960326    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d5gjj"
Oct 30 11:30:14.995927 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.995881    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kpjxg" event={"ID":"5e534504-8704-4af2-8d4c-a2bd32767adb","Type":"ContainerStarted","Data":"a1c7bc5a5b19e9915e6222019fef4b4da1eaaf99f67303611cfe4a2309abe6ba"}
Oct 30 11:30:14.997564 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.997534    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth" event={"ID":"232e3b46-20e1-450a-9d33-13d6c271ea30","Type":"ContainerStarted","Data":"0509516c0449c2fd1af45a74d1b2c2af7edacca8ef9eea7b37011a434c9ffd4b"}
Oct 30 11:30:14.999660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.999635    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8" event={"ID":"9c34812b-7d7c-4743-bcea-78110c211de4","Type":"ContainerStarted","Data":"4a368b62512d77acc98ef1a95637b21c456d6a3f00b0dc92faa55bae9e239491"}
Oct 30 11:30:14.999766 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:14.999678    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8" event={"ID":"9c34812b-7d7c-4743-bcea-78110c211de4","Type":"ContainerStarted","Data":"c9928f1b7c5e6133df748738e2511ec6d45a7e2b78f374ff062662ceba25c357"}
Oct 30 11:30:15.000242 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:15.000223    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:15.048490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:15.048440    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8" podStartSLOduration=2.048421996 podStartE2EDuration="2.048421996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:15.048054751 +0000 UTC m=+14.268339663" watchObservedRunningTime="2026-10-30 11:30:15.048421996 +0000 UTC m=+14.268706903"
Oct 30 11:30:15.312668 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:15.312638    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d5gjj_38b69f25-5947-4982-8a04-bc09d51a87b3/dns/0.log"
Oct 30 11:30:15.453395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:15.453363    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d5gjj_38b69f25-5947-4982-8a04-bc09d51a87b3/kube-rbac-proxy/0.log"
Oct 30 11:30:16.004513 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:16.004472    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth" event={"ID":"232e3b46-20e1-450a-9d33-13d6c271ea30","Type":"ContainerStarted","Data":"18983aad23713f1283c0335cf8e1a9e75fa1e8b4ba378b034f2b13861f2d53a7"}
Oct 30 11:30:16.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:16.044893    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth" podStartSLOduration=2.152003734 podStartE2EDuration="3.044878351s" totalImagesPullingTime="892.874617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:16.044298178 +0000 UTC m=+15.264583089" watchObservedRunningTime="2026-10-30 11:30:16.044878351 +0000 UTC m=+15.265163262"
Oct 30 11:30:16.453373 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:16.453325    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6llc_ef3a0a4d-2922-4948-af17-cf737a3f6273/dns-node-resolver/0.log"
Oct 30 11:30:16.653608 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:16.653579    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58b589f7fb-84kq8_9c34812b-7d7c-4743-bcea-78110c211de4/registry/0.log"
Oct 30 11:30:17.007760 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:17.007728    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:17.013408 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:17.013384    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vhbth"
Oct 30 11:30:17.254071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:17.254040    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bfdd_cb76f4db-7292-45d7-8780-f5e14eedbfd3/node-ca/0.log"
Oct 30 11:30:18.053640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.053611    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvpqd_96e3a31c-a58f-42b7-a29d-3f41ef0d3435/serve-healthcheck-canary/0.log"
Oct 30 11:30:18.310293 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.310216    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"]
Oct 30 11:30:18.313700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.313677    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.343650 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.343629    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 11:30:18.344060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.343939    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 11:30:18.344060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.343996    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 11:30:18.344060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.344025    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 11:30:18.344060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.344041    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hbnhw\""
Oct 30 11:30:18.344060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.343999    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 11:30:18.356653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.356627    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"]
Oct 30 11:30:18.448492 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.448457    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.448492 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.448499    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.448697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.448529    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wk8l\" (UniqueName: \"kubernetes.io/projected/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-kube-api-access-5wk8l\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.448697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.448615    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.550093 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.550063    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.550239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.550103    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.550239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.550129    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wk8l\" (UniqueName: \"kubernetes.io/projected/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-kube-api-access-5wk8l\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.550239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.550157    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.550996 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.550947    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.552985 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.552963    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.553143 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.553120    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.575071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.575019    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wk8l\" (UniqueName: \"kubernetes.io/projected/d1f6c3ab-4b41-4c69-a832-d98dd617ce26-kube-api-access-5wk8l\") pod \"prometheus-operator-9cd89b74c-bkwfj\" (UID: \"d1f6c3ab-4b41-4c69-a832-d98dd617ce26\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.628834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.628806    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"
Oct 30 11:30:18.932444 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:18.932422    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj"]
Oct 30 11:30:18.934099 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:18.934057    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f6c3ab_4b41_4c69_a832_d98dd617ce26.slice/crio-014fcdb4db18d3d21afcf33a0fe701964139922125c6384783acfba8c35bce37 WatchSource:0}: Error finding container 014fcdb4db18d3d21afcf33a0fe701964139922125c6384783acfba8c35bce37: Status 404 returned error can't find the container with id 014fcdb4db18d3d21afcf33a0fe701964139922125c6384783acfba8c35bce37
Oct 30 11:30:19.016256 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:19.016218    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj" event={"ID":"d1f6c3ab-4b41-4c69-a832-d98dd617ce26","Type":"ContainerStarted","Data":"014fcdb4db18d3d21afcf33a0fe701964139922125c6384783acfba8c35bce37"}
Oct 30 11:30:21.025274 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:21.025236    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj" event={"ID":"d1f6c3ab-4b41-4c69-a832-d98dd617ce26","Type":"ContainerStarted","Data":"174c16ff7d0c45cdfd730992d07cd0ebfd06a03263d65b9a28a51fba3926d1bf"}
Oct 30 11:30:21.025853 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:21.025296    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj" event={"ID":"d1f6c3ab-4b41-4c69-a832-d98dd617ce26","Type":"ContainerStarted","Data":"71afe41de908dd813b3563e213e87ebcc29aabdbf2d6fd183a0d95b82b336bc9"}
Oct 30 11:30:21.068440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:21.068384    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bkwfj" podStartSLOduration=1.916157213 podStartE2EDuration="3.068365479s" totalImagesPullingTime="1.152208266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:21.06832182 +0000 UTC m=+20.288606730" watchObservedRunningTime="2026-10-30 11:30:21.068365479 +0000 UTC m=+20.288650384"
Oct 30 11:30:22.952076 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.951999    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"]
Oct 30 11:30:22.955719 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.955696    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:22.975079 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.975055    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4wb85\""
Oct 30 11:30:22.987042 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.987013    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 11:30:22.987407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.987388    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 11:30:22.988371 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:22.988351    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"]
Oct 30 11:30:23.024092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.024066    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l8sqn"]
Oct 30 11:30:23.027055 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.027035    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.052804 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.052776    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 11:30:23.052804 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.052793    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 11:30:23.052963 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.052859    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frflv\""
Oct 30 11:30:23.085040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.085014    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7kd\" (UniqueName: \"kubernetes.io/projected/ca9eb8a5-7274-4024-841d-c26780902023-kube-api-access-qn7kd\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.085191 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.085072    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.085191 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.085156    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.085312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.085223    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9eb8a5-7274-4024-841d-c26780902023-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.134252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.134220    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 11:30:23.186600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186551    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7kd\" (UniqueName: \"kubernetes.io/projected/ca9eb8a5-7274-4024-841d-c26780902023-kube-api-access-qn7kd\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.186749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186606    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-tls\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.186749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186636    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6mrn\" (UniqueName: \"kubernetes.io/projected/53c0c70e-e0f3-487b-9699-49f3cc352530-kube-api-access-b6mrn\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.186852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186789    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.186852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186819    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.186852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186844    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-wtmp\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.186981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186879    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-textfile\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.186981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.186920    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9eb8a5-7274-4024-841d-c26780902023-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.187355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.187301    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-sys\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.187459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.187373    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-metrics-client-ca\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.187459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.187417    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-root\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.187571 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.187518    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.187729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.187705    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.188126 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.188103    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca9eb8a5-7274-4024-841d-c26780902023-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.189679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.189654    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.189797 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.189676    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca9eb8a5-7274-4024-841d-c26780902023-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.212791 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.212742    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7kd\" (UniqueName: \"kubernetes.io/projected/ca9eb8a5-7274-4024-841d-c26780902023-kube-api-access-qn7kd\") pod \"openshift-state-metrics-7bd7489957-tg95x\" (UID: \"ca9eb8a5-7274-4024-841d-c26780902023\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.272769 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.272736    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"
Oct 30 11:30:23.288427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288397    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-sys\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288446    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-metrics-client-ca\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288478    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-root\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288507    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288544    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288574    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-tls\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288597    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6mrn\" (UniqueName: \"kubernetes.io/projected/53c0c70e-e0f3-487b-9699-49f3cc352530-kube-api-access-b6mrn\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288627    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-wtmp\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.288734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288657    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-textfile\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288935    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-textfile\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.288964    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-sys\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.289019    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-root\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.289083    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-wtmp\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289649 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.289574    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.289742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.289666    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53c0c70e-e0f3-487b-9699-49f3cc352530-metrics-client-ca\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.291703 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.291660    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.291925 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.291907    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53c0c70e-e0f3-487b-9699-49f3cc352530-node-exporter-tls\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.313565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.313541    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6mrn\" (UniqueName: \"kubernetes.io/projected/53c0c70e-e0f3-487b-9699-49f3cc352530-kube-api-access-b6mrn\") pod \"node-exporter-l8sqn\" (UID: \"53c0c70e-e0f3-487b-9699-49f3cc352530\") " pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.345690 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.345654    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8sqn"
Oct 30 11:30:23.361616 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:23.361584    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c0c70e_e0f3_487b_9699_49f3cc352530.slice/crio-9cded43e09024d1ddec2c7f3e081097bcb2feb1d2854e7f27a1cef426a1ac1f6 WatchSource:0}: Error finding container 9cded43e09024d1ddec2c7f3e081097bcb2feb1d2854e7f27a1cef426a1ac1f6: Status 404 returned error can't find the container with id 9cded43e09024d1ddec2c7f3e081097bcb2feb1d2854e7f27a1cef426a1ac1f6
Oct 30 11:30:23.588538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:23.588507    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x"]
Oct 30 11:30:23.589848 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:30:23.589817    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9eb8a5_7274_4024_841d_c26780902023.slice/crio-6ee38db1fe184a19ca470c22c2151e4bce2cb0858ee4746687ae74859bc47aa4 WatchSource:0}: Error finding container 6ee38db1fe184a19ca470c22c2151e4bce2cb0858ee4746687ae74859bc47aa4: Status 404 returned error can't find the container with id 6ee38db1fe184a19ca470c22c2151e4bce2cb0858ee4746687ae74859bc47aa4
Oct 30 11:30:24.038302 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:24.038036    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8sqn" event={"ID":"53c0c70e-e0f3-487b-9699-49f3cc352530","Type":"ContainerStarted","Data":"9cded43e09024d1ddec2c7f3e081097bcb2feb1d2854e7f27a1cef426a1ac1f6"}
Oct 30 11:30:24.040275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:24.040165    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x" event={"ID":"ca9eb8a5-7274-4024-841d-c26780902023","Type":"ContainerStarted","Data":"cab9220f41a9d69993444969c827f5b2dc3470da809ba4d5f7617796338f1fef"}
Oct 30 11:30:24.040275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:24.040222    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x" event={"ID":"ca9eb8a5-7274-4024-841d-c26780902023","Type":"ContainerStarted","Data":"97b92dd10e95c0d54243fd3e08049f91f4336de2403c5935a8a8d76f277871ee"}
Oct 30 11:30:24.040275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:24.040241    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x" event={"ID":"ca9eb8a5-7274-4024-841d-c26780902023","Type":"ContainerStarted","Data":"6ee38db1fe184a19ca470c22c2151e4bce2cb0858ee4746687ae74859bc47aa4"}
Oct 30 11:30:25.046476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:25.046312    4453 generic.go:356] "Generic (PLEG): container finished" podID="53c0c70e-e0f3-487b-9699-49f3cc352530" containerID="41eb080cde778f4f579564b0e75dee3dda14f50c557e0415fc55046c7a1561f4" exitCode=0
Oct 30 11:30:25.046476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:25.046409    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8sqn" event={"ID":"53c0c70e-e0f3-487b-9699-49f3cc352530","Type":"ContainerDied","Data":"41eb080cde778f4f579564b0e75dee3dda14f50c557e0415fc55046c7a1561f4"}
Oct 30 11:30:25.049414 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:25.049346    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x" event={"ID":"ca9eb8a5-7274-4024-841d-c26780902023","Type":"ContainerStarted","Data":"b0f6fcf0384c948b89532decca4d5cbaaf2ef384872fb0661e4848798c2d6c66"}
Oct 30 11:30:25.145318 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:25.145249    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-tg95x" podStartSLOduration=2.16580551 podStartE2EDuration="3.145231479s" totalImagesPullingTime="979.425969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:25.144079153 +0000 UTC m=+24.364364064" watchObservedRunningTime="2026-10-30 11:30:25.145231479 +0000 UTC m=+24.365516389"
Oct 30 11:30:26.056547 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:26.056508    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8sqn" event={"ID":"53c0c70e-e0f3-487b-9699-49f3cc352530","Type":"ContainerStarted","Data":"9aa02cb8c4921d9a8f90a08fb72ea3c0e7c1c5c3e9d052a453b1872189e2a174"}
Oct 30 11:30:26.057017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:26.056579    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8sqn" event={"ID":"53c0c70e-e0f3-487b-9699-49f3cc352530","Type":"ContainerStarted","Data":"4e92905975fbb5a3805ed6f3e3790da419235a7f836911782604217f0ac2d08f"}
Oct 30 11:30:26.125994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:26.125931    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l8sqn" podStartSLOduration=3.183675362 podStartE2EDuration="4.125917343s" totalImagesPullingTime="862.102102ms" totalInitContainerRuntime="80.139879ms" isStatefulPod=true podCreationTimestamp="2026-10-30 11:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:26.113542735 +0000 UTC m=+25.333827646" watchObservedRunningTime="2026-10-30 11:30:26.125917343 +0000 UTC m=+25.346202250"
Oct 30 11:30:31.081623 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:31.081583    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-kpjxg" event={"ID":"5e534504-8704-4af2-8d4c-a2bd32767adb","Type":"ContainerStarted","Data":"03aa87464854e682654bb09bfc0c01f03b350e936f5d907990a8891c1dfbe1c3"}
Oct 30 11:30:31.082112 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:31.081875    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:31.097813 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:31.097785    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-kpjxg"
Oct 30 11:30:31.147873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:31.147824    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-kpjxg" podStartSLOduration=2.348578072 podStartE2EDuration="18.147811888s" totalImagesPullingTime="15.799233816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:30:31.135650669 +0000 UTC m=+30.355935579" watchObservedRunningTime="2026-10-30 11:30:31.147811888 +0000 UTC m=+30.368096799"
Oct 30 11:30:33.356450 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:33.356416    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qq4wf"
Oct 30 11:30:37.014020 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:30:37.013988    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58b589f7fb-84kq8"
Oct 30 11:30:37.572701 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:30:37.572659    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:31:07.568890 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:31:07.568850    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:31:37.570018 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:31:37.569972    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:32:07.568441 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:32:07.568398    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:32:37.568671 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:32:37.568590    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:33:07.568677 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:33:07.568642    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:33:37.568589 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:33:37.568546    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:34:07.569504 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:34:07.569465    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:34:37.569275 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:34:37.569237    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:35:03.260506 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.260466    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:35:03.263440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.263422    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.283490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.283471    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 11:35:03.283604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.283506    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 11:35:03.283604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.283597    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 11:35:03.283679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.283595    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dn7s8\""
Oct 30 11:35:03.295566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.295541    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 11:35:03.295674 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.295587    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 11:35:03.295674 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.295649    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 11:35:03.301298 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.301275    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:35:03.311701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311676    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.311807 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311718    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktbg\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.311807 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311753    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.311807 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311777    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.311954 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311850    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.311954 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311918    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.312059 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.311995    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412641 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412609    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412641 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412648    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412772    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412810    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktbg\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412831    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.412865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412847    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.413141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.412869    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.413390 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.413367    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.415055 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.415028    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.415155 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.415141    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.415486 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.415466    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.415568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.415466    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.441671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.441649    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.442232 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.442208    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktbg\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\") pod \"istiod-openshift-gateway-7876bfcb74-cgqsn\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.574498 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.574421    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:03.889104 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.889075    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:35:03.890030 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:35:03.890002    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10ee7f7_6d8d_4cd6_a228_edccbd33c8d9.slice/crio-a245b4a4375dbf631ed3212d25abbec4c4be17462ac2801b28077201d203271b WatchSource:0}: Error finding container a245b4a4375dbf631ed3212d25abbec4c4be17462ac2801b28077201d203271b: Status 404 returned error can't find the container with id a245b4a4375dbf631ed3212d25abbec4c4be17462ac2801b28077201d203271b
Oct 30 11:35:03.892050 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.892031    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:35:03.945518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:03.945481    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" event={"ID":"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9","Type":"ContainerStarted","Data":"a245b4a4375dbf631ed3212d25abbec4c4be17462ac2801b28077201d203271b"}
Oct 30 11:35:06.401048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:06.401015    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:35:06.401357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:06.401083    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:35:06.957604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:06.957570    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" event={"ID":"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9","Type":"ContainerStarted","Data":"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"}
Oct 30 11:35:06.957775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:06.957702    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:07.020842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:07.020787    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" podStartSLOduration=1.512706814 podStartE2EDuration="4.020773358s" totalImagesPullingTime="2.508066544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:07.007862745 +0000 UTC m=+306.228147657" watchObservedRunningTime="2026-10-30 11:35:07.020773358 +0000 UTC m=+306.241058262"
Oct 30 11:35:07.569490 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:35:07.569457    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:35:07.962205 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:07.962168    4453 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cgqsn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 11:35:07.962376 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:07.962244    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 11:35:10.961373 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:10.961320    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:35:37.569388 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:35:37.569284    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:35:48.693665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.693628    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"]
Oct 30 11:35:48.698755 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.698731    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.731852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.731820    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"]
Oct 30 11:35:48.799578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799542    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 11:35:48.799807 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799786    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 11:35:48.799863 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799846    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 11:35:48.799897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799850    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 11:35:48.799949 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799935    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bjbjc\""
Oct 30 11:35:48.799981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.799964    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 11:35:48.858181 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858152    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.858181 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858182    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.858369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858201    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.858369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858257    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wwq\" (UniqueName: \"kubernetes.io/projected/7ed0c0dc-8474-4beb-86c8-5462744c14ef-kube-api-access-p4wwq\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.858435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858365    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.858435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.858403    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959595    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959631    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959690    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959717    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959739    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.959859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.959765    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wwq\" (UniqueName: \"kubernetes.io/projected/7ed0c0dc-8474-4beb-86c8-5462744c14ef-kube-api-access-p4wwq\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.960263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.960233    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-manifests\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.962050 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.962024    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed0c0dc-8474-4beb-86c8-5462744c14ef-tmp\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.962168 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.962078    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.962273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.962251    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.962353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.962321    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed0c0dc-8474-4beb-86c8-5462744c14ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:48.986426 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:48.986408    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wwq\" (UniqueName: \"kubernetes.io/projected/7ed0c0dc-8474-4beb-86c8-5462744c14ef-kube-api-access-p4wwq\") pod \"opendatahub-operator-controller-manager-7f5b5d7667-wmq8s\" (UID: \"7ed0c0dc-8474-4beb-86c8-5462744c14ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:49.012836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:49.012812    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:49.365044 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:49.365013    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"]
Oct 30 11:35:49.368190 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:35:49.368165    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed0c0dc_8474_4beb_86c8_5462744c14ef.slice/crio-f5d0045ea12c11a6fa2fd78657366b8105aff98fa7c93e53528f07fac131f8df WatchSource:0}: Error finding container f5d0045ea12c11a6fa2fd78657366b8105aff98fa7c93e53528f07fac131f8df: Status 404 returned error can't find the container with id f5d0045ea12c11a6fa2fd78657366b8105aff98fa7c93e53528f07fac131f8df
Oct 30 11:35:50.099666 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:50.099627    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s" event={"ID":"7ed0c0dc-8474-4beb-86c8-5462744c14ef","Type":"ContainerStarted","Data":"f5d0045ea12c11a6fa2fd78657366b8105aff98fa7c93e53528f07fac131f8df"}
Oct 30 11:35:53.111289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:53.111251    4453 generic.go:356] "Generic (PLEG): container finished" podID="7ed0c0dc-8474-4beb-86c8-5462744c14ef" containerID="a936a5b4abd9d71f110fa42823c49bb590989b4a1af0409b42ed8ecdb21e8bf0" exitCode=0
Oct 30 11:35:53.111785 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:53.111297    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s" event={"ID":"7ed0c0dc-8474-4beb-86c8-5462744c14ef","Type":"ContainerDied","Data":"a936a5b4abd9d71f110fa42823c49bb590989b4a1af0409b42ed8ecdb21e8bf0"}
Oct 30 11:35:54.117772 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:54.117730    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s" event={"ID":"7ed0c0dc-8474-4beb-86c8-5462744c14ef","Type":"ContainerStarted","Data":"5b28d63f6577f4b09dc490c410a7666a5da5e69415d3f59a44c8d1c1d5117c85"}
Oct 30 11:35:54.118258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:54.117881    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:35:54.203948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:35:54.203898    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s" podStartSLOduration=2.582697688 podStartE2EDuration="6.203885077s" totalImagesPullingTime="3.146869798s" totalInitContainerRuntime="474.317591ms" isStatefulPod=false podCreationTimestamp="2026-10-30 11:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:35:54.194200763 +0000 UTC m=+353.414485674" watchObservedRunningTime="2026-10-30 11:35:54.203885077 +0000 UTC m=+353.424169987"
Oct 30 11:36:05.123910 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:05.123873    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5b5d7667-wmq8s"
Oct 30 11:36:07.569003 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:36:07.568964    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:36:28.706204 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.706163    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-p6rqz"]
Oct 30 11:36:28.714000 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.713979    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.736423 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.736391    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f404404b-d627-461a-b827-1e4e5bafa0fc-tls-certs\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.736559 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.736442    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f404404b-d627-461a-b827-1e4e5bafa0fc-tmp\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.736559 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.736514    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pm8\" (UniqueName: \"kubernetes.io/projected/f404404b-d627-461a-b827-1e4e5bafa0fc-kube-api-access-x5pm8\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.740942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.740908    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 11:36:28.741087 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.740963    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 11:36:28.741087 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.741014    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v9czc\""
Oct 30 11:36:28.742198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.742171    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-p6rqz"]
Oct 30 11:36:28.837656 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.837614    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f404404b-d627-461a-b827-1e4e5bafa0fc-tls-certs\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.837831 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.837664    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f404404b-d627-461a-b827-1e4e5bafa0fc-tmp\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.837831 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.837685    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pm8\" (UniqueName: \"kubernetes.io/projected/f404404b-d627-461a-b827-1e4e5bafa0fc-kube-api-access-x5pm8\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.840000 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.839969    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f404404b-d627-461a-b827-1e4e5bafa0fc-tmp\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.840231 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.840213    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f404404b-d627-461a-b827-1e4e5bafa0fc-tls-certs\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:28.858593 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:28.858566    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pm8\" (UniqueName: \"kubernetes.io/projected/f404404b-d627-461a-b827-1e4e5bafa0fc-kube-api-access-x5pm8\") pod \"kube-auth-proxy-775b454769-p6rqz\" (UID: \"f404404b-d627-461a-b827-1e4e5bafa0fc\") " pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:29.026197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:29.026119    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz"
Oct 30 11:36:29.339289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:29.339205    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-775b454769-p6rqz"]
Oct 30 11:36:29.343226 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:36:29.343189    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf404404b_d627_461a_b827_1e4e5bafa0fc.slice/crio-f75eefd1f1bda1aa94ae215ec15c91907ef089647f8c801db070701fccf575f0 WatchSource:0}: Error finding container f75eefd1f1bda1aa94ae215ec15c91907ef089647f8c801db070701fccf575f0: Status 404 returned error can't find the container with id f75eefd1f1bda1aa94ae215ec15c91907ef089647f8c801db070701fccf575f0
Oct 30 11:36:30.244906 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:30.244865    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz" event={"ID":"f404404b-d627-461a-b827-1e4e5bafa0fc","Type":"ContainerStarted","Data":"f75eefd1f1bda1aa94ae215ec15c91907ef089647f8c801db070701fccf575f0"}
Oct 30 11:36:31.469720 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.469684    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"]
Oct 30 11:36:31.473190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.473168    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.496278 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496250    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-stfdr\""
Oct 30 11:36:31.496447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496306    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 11:36:31.496447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496314    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 11:36:31.496447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496257    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 11:36:31.496447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496363    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 11:36:31.496447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.496410    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 11:36:31.510492 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.510468    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"]
Oct 30 11:36:31.561130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.561098    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-metrics-certs\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.561269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.561138    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa723a-6939-41e8-84cb-159af9288da1-manager-config\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.561315 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.561281    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqpl\" (UniqueName: \"kubernetes.io/projected/2cfa723a-6939-41e8-84cb-159af9288da1-kube-api-access-ltqpl\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.561378 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.561321    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-cert\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.666207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.665680    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-metrics-certs\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.666207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.665727    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa723a-6939-41e8-84cb-159af9288da1-manager-config\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.666207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.665786    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqpl\" (UniqueName: \"kubernetes.io/projected/2cfa723a-6939-41e8-84cb-159af9288da1-kube-api-access-ltqpl\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.666207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.665814    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-cert\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.666789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.666767    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2cfa723a-6939-41e8-84cb-159af9288da1-manager-config\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.668772 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.668746    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-metrics-certs\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.668885 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.668789    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cfa723a-6939-41e8-84cb-159af9288da1-cert\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.711688 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.711656    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqpl\" (UniqueName: \"kubernetes.io/projected/2cfa723a-6939-41e8-84cb-159af9288da1-kube-api-access-ltqpl\") pod \"jobset-controller-manager-7bcfbd779f-sq2j8\" (UID: \"2cfa723a-6939-41e8-84cb-159af9288da1\") " pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:31.787946 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:31.787922    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:32.101608 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:32.101506    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"]
Oct 30 11:36:32.105949 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:36:32.105904    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfa723a_6939_41e8_84cb_159af9288da1.slice/crio-81205e46c1956438c15461a47ee9d445530da3d9a71ea1eddee018d8614b9079 WatchSource:0}: Error finding container 81205e46c1956438c15461a47ee9d445530da3d9a71ea1eddee018d8614b9079: Status 404 returned error can't find the container with id 81205e46c1956438c15461a47ee9d445530da3d9a71ea1eddee018d8614b9079
Oct 30 11:36:32.254508 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:32.254473    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz" event={"ID":"f404404b-d627-461a-b827-1e4e5bafa0fc","Type":"ContainerStarted","Data":"6af84368eec77c70e44fdd0e2ad703669090dcf9049965cd8d9378fb7f39c080"}
Oct 30 11:36:32.255689 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:32.255653    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8" event={"ID":"2cfa723a-6939-41e8-84cb-159af9288da1","Type":"ContainerStarted","Data":"81205e46c1956438c15461a47ee9d445530da3d9a71ea1eddee018d8614b9079"}
Oct 30 11:36:32.291713 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:32.291661    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-775b454769-p6rqz" podStartSLOduration=1.861473926 podStartE2EDuration="4.291646328s" totalImagesPullingTime="2.430172402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:32.290386358 +0000 UTC m=+391.510671270" watchObservedRunningTime="2026-10-30 11:36:32.291646328 +0000 UTC m=+391.511931237"
Oct 30 11:36:35.268615 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:35.268577    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8" event={"ID":"2cfa723a-6939-41e8-84cb-159af9288da1","Type":"ContainerStarted","Data":"db88b55de0768cf1317c1044b45e7ddc0c4cb92f1afa7206be822d6ef205f31f"}
Oct 30 11:36:35.269022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:35.268722    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:36:35.311056 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:35.311008    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8" podStartSLOduration=1.80630025 podStartE2EDuration="4.310995592s" totalImagesPullingTime="2.504695342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:36:35.310272745 +0000 UTC m=+394.530557665" watchObservedRunningTime="2026-10-30 11:36:35.310995592 +0000 UTC m=+394.531280502"
Oct 30 11:36:37.568635 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:36:37.568598    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:36:46.277798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:36:46.277728    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7bcfbd779f-sq2j8"
Oct 30 11:37:07.568850 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:37:07.568812    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:37:30.155673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.155639    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-sthrh"]
Oct 30 11:37:30.158789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.158767    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.174921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.174892    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 11:37:30.181233 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.181217    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 11:37:30.185258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.185238    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-sthrh"]
Oct 30 11:37:30.205482 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.205462    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h8p4f\""
Oct 30 11:37:30.205657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.205642    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 11:37:30.205741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.205729    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 11:37:30.206013 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.205990    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-metrics-certs\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.206065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.206047    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-cert\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.206163 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.206139    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzt5\" (UniqueName: \"kubernetes.io/projected/0f674187-2f7c-4153-9d68-b16a094df612-kube-api-access-6fzt5\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.307149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.307112    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-cert\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.307149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.307151    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzt5\" (UniqueName: \"kubernetes.io/projected/0f674187-2f7c-4153-9d68-b16a094df612-kube-api-access-6fzt5\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.307388 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.307214    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-metrics-certs\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.309521 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.309491    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-cert\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.309633 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.309587    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f674187-2f7c-4153-9d68-b16a094df612-metrics-certs\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.329428 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.329388    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzt5\" (UniqueName: \"kubernetes.io/projected/0f674187-2f7c-4153-9d68-b16a094df612-kube-api-access-6fzt5\") pod \"odh-model-controller-769875db56-sthrh\" (UID: \"0f674187-2f7c-4153-9d68-b16a094df612\") " pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.470294 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.470217    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:30.762893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:30.762804    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-sthrh"]
Oct 30 11:37:30.766910 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:37:30.766881    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f674187_2f7c_4153_9d68_b16a094df612.slice/crio-b24be01852b8e1986464384b065af5bc4c08e03e5de334af2429cfd18718f3ea WatchSource:0}: Error finding container b24be01852b8e1986464384b065af5bc4c08e03e5de334af2429cfd18718f3ea: Status 404 returned error can't find the container with id b24be01852b8e1986464384b065af5bc4c08e03e5de334af2429cfd18718f3ea
Oct 30 11:37:31.463104 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:31.463065    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-sthrh" event={"ID":"0f674187-2f7c-4153-9d68-b16a094df612","Type":"ContainerStarted","Data":"b24be01852b8e1986464384b065af5bc4c08e03e5de334af2429cfd18718f3ea"}
Oct 30 11:37:33.473346 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:33.473292    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-sthrh" event={"ID":"0f674187-2f7c-4153-9d68-b16a094df612","Type":"ContainerStarted","Data":"1339d873525616b93e27dfe814ce1a460b4beb727dba492b58342fe6a5a558d3"}
Oct 30 11:37:33.473743 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:33.473486    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:33.518464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:33.518412    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-sthrh" podStartSLOduration=0.988636398 podStartE2EDuration="3.518398765s" totalImagesPullingTime="2.529762367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:37:33.517783781 +0000 UTC m=+452.738068691" watchObservedRunningTime="2026-10-30 11:37:33.518398765 +0000 UTC m=+452.738683674"
Oct 30 11:37:33.939190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:33.939147    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:37:33.939424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:33.939389    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerName="discovery" containerID="cri-o://b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1" gracePeriod=30
Oct 30 11:37:34.183004 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.182977    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:37:34.244299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244227    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244262    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244287    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244560 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244318    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244560 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244358    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244560 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244432    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244560 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244467    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\") pod \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\" (UID: \"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9\") "
Oct 30 11:37:34.244769 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.244686    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:37:34.246753 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.246725    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "kube-api-access-hktbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:37:34.246882 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.246727    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:37:34.246882 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.246847    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:37:34.246974 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.246897    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:37:34.246974 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.246927    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:37:34.247038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.247016    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig" pod "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" (UID: "d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:37:34.345269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345236    4453 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-local-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345265    4453 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-kubeconfig\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345275    4453 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-dns-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345284    4453 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-csr-ca-configmap\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345293    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345302    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktbg\" (UniqueName: \"kubernetes.io/projected/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-kube-api-access-hktbg\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.345520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.345310    4453 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9-cacerts\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:37:34.477899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.477867    4453 generic.go:356] "Generic (PLEG): container finished" podID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerID="b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1" exitCode=0
Oct 30 11:37:34.478326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.477915    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"
Oct 30 11:37:34.478326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.477966    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" event={"ID":"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9","Type":"ContainerDied","Data":"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"}
Oct 30 11:37:34.478326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.478015    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn" event={"ID":"d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9","Type":"ContainerDied","Data":"a245b4a4375dbf631ed3212d25abbec4c4be17462ac2801b28077201d203271b"}
Oct 30 11:37:34.478326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.478054    4453 scope.go:122] "RemoveContainer" containerID="b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"
Oct 30 11:37:34.486718 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.486698    4453 scope.go:122] "RemoveContainer" containerID="b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"
Oct 30 11:37:34.487002 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:37:34.486980    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1\": container with ID starting with b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1 not found: ID does not exist" containerID="b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"
Oct 30 11:37:34.487057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.487011    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1"} err="failed to get container status \"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1\": rpc error: code = NotFound desc = could not find container \"b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1\": container with ID starting with b9d2770e847259287636ba4018657d568e43ca20f87ca7320d7e9fdfe23ba3c1 not found: ID does not exist"
Oct 30 11:37:34.523880 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.523812    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:37:34.533684 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.533659    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cgqsn"]
Oct 30 11:37:34.908898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:34.908868    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" path="/var/lib/kubelet/pods/d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9/volumes"
Oct 30 11:37:37.569233 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:37:37.569196    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:37:44.480945 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:44.480910    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-sthrh"
Oct 30 11:37:57.851877 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.851842    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:37:57.852251 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.852121    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerName="discovery"
Oct 30 11:37:57.852251 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.852132    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerName="discovery"
Oct 30 11:37:57.852251 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.852184    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10ee7f7-6d8d-4cd6-a228-edccbd33c8d9" containerName="discovery"
Oct 30 11:37:57.855206 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.855183    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:57.877275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.877256    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 11:37:57.877397 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.877298    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 11:37:57.877611 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.877594    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 11:37:57.877611 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.877595    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cf29m\""
Oct 30 11:37:57.877777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.877661    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 11:37:57.880185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.880163    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:37:57.916759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.916731    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:57.916877 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.916762    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:57.916877 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.916863    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxx7p\" (UniqueName: \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:57.916982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:57.916920    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.017610 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.017585    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.017723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.017611    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.017723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.017654    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxx7p\" (UniqueName: \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.017723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.017690    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.018305 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.018287    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.019952 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.019932    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.020059 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.020041    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.038383 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.038358    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxx7p\" (UniqueName: \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\") pod \"maas-api-5c59648cf5-vjj4t\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") " pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.166705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.166670    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:37:58.460232 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.460051    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:37:58.463011 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:37:58.462987    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9912f4e3_2e4b_4c7e_a04c_24260e5c301f.slice/crio-efaa5def97fb61dd13b2a503472abeed9509ed7d78c03dcb01897a017994b6e1 WatchSource:0}: Error finding container efaa5def97fb61dd13b2a503472abeed9509ed7d78c03dcb01897a017994b6e1: Status 404 returned error can't find the container with id efaa5def97fb61dd13b2a503472abeed9509ed7d78c03dcb01897a017994b6e1
Oct 30 11:37:58.570163 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:37:58.570132    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" event={"ID":"9912f4e3-2e4b-4c7e-a04c-24260e5c301f","Type":"ContainerStarted","Data":"efaa5def97fb61dd13b2a503472abeed9509ed7d78c03dcb01897a017994b6e1"}
Oct 30 11:38:01.583815 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:01.583777    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" event={"ID":"9912f4e3-2e4b-4c7e-a04c-24260e5c301f","Type":"ContainerStarted","Data":"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"}
Oct 30 11:38:01.584223 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:01.583899    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:38:01.621464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:01.621423    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" podStartSLOduration=2.3005741 podStartE2EDuration="4.621411744s" totalImagesPullingTime="2.320837644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:01.621261831 +0000 UTC m=+480.841546741" watchObservedRunningTime="2026-10-30 11:38:01.621411744 +0000 UTC m=+480.841696651"
Oct 30 11:38:07.569066 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:38:07.569023    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:38:07.592040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:07.592015    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:38:08.399987 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.399953    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xh6qt"]
Oct 30 11:38:08.404573 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.404550    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.422708 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.422686    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7lq8x\""
Oct 30 11:38:08.423780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.423762    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xh6qt"]
Oct 30 11:38:08.498127 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.498101    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crgx\" (UniqueName: \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\") pod \"authorino-6f6944bcdd-xh6qt\" (UID: \"57d63407-3c94-4667-ac36-ea1d26bb3775\") " pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.504544 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.504518    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 11:38:08.504630 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.504617    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 11:38:08.583859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.583828    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xh6qt"]
Oct 30 11:38:08.584225 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:38:08.583995    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5crgx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-xh6qt" podUID="57d63407-3c94-4667-ac36-ea1d26bb3775"
Oct 30 11:38:08.599275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.599245    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crgx\" (UniqueName: \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\") pod \"authorino-6f6944bcdd-xh6qt\" (UID: \"57d63407-3c94-4667-ac36-ea1d26bb3775\") " pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.605973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.605954    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.609781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.609760    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"]
Oct 30 11:38:08.612282 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.612264    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.613044 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.613030    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.615130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.615112    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8qghd"]
Oct 30 11:38:08.617667 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.617650    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.628427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.628406    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crgx\" (UniqueName: \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\") pod \"authorino-6f6944bcdd-xh6qt\" (UID: \"57d63407-3c94-4667-ac36-ea1d26bb3775\") " pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:08.630603 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.630589    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 11:38:08.638214 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.638195    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"]
Oct 30 11:38:08.640960 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.640940    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8qghd"]
Oct 30 11:38:08.700514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700431    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\" (UniqueName: \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\") pod \"57d63407-3c94-4667-ac36-ea1d26bb3775\" (UID: \"57d63407-3c94-4667-ac36-ea1d26bb3775\") "
Oct 30 11:38:08.700680 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700564    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-maas-api-tls\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.700680 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700588    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4hn\" (UniqueName: \"kubernetes.io/projected/c6b9e879-cbef-433f-8388-15fc39e8b78c-kube-api-access-9r4hn\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.700680 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700622    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-metrics-certs\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.700842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700757    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9e879-cbef-433f-8388-15fc39e8b78c-openshift-service-ca\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.700842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700785    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.700842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.700806    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pxf\" (UniqueName: \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.702672 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.702651    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx" pod "57d63407-3c94-4667-ac36-ea1d26bb3775" (UID: "57d63407-3c94-4667-ac36-ea1d26bb3775"). InnerVolumeSpecName "kube-api-access-5crgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:38:08.795568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.795532    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8qghd"]
Oct 30 11:38:08.795745 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:38:08.795717    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-76pxf tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-8qghd" podUID="879ba9d2-9e13-4282-a477-82fc5286ec60"
Oct 30 11:38:08.801533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801506    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-metrics-certs\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.801661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801554    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9e879-cbef-433f-8388-15fc39e8b78c-openshift-service-ca\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.801712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801664    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.801712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801697    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pxf\" (UniqueName: \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.801791 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801724    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-maas-api-tls\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.801791 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801746    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4hn\" (UniqueName: \"kubernetes.io/projected/c6b9e879-cbef-433f-8388-15fc39e8b78c-kube-api-access-9r4hn\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.801873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.801798    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5crgx\" (UniqueName: \"kubernetes.io/projected/57d63407-3c94-4667-ac36-ea1d26bb3775-kube-api-access-5crgx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:08.802255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.802229    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b9e879-cbef-433f-8388-15fc39e8b78c-openshift-service-ca\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.804069 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.804042    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-metrics-certs\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.804153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.804069    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b9e879-cbef-433f-8388-15fc39e8b78c-maas-api-tls\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.804153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.804104    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.826024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.826002    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pxf\" (UniqueName: \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\") pod \"authorino-6889cbd8c4-8qghd\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") " pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:08.827626 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.827603    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4hn\" (UniqueName: \"kubernetes.io/projected/c6b9e879-cbef-433f-8388-15fc39e8b78c-kube-api-access-9r4hn\") pod \"maas-api-c6d5fb647-8rlbz\" (UID: \"c6b9e879-cbef-433f-8388-15fc39e8b78c\") " pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:08.834149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.834126    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:38:08.837844 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.837818    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:08.857190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.857169    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:38:08.902424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.902394    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmck6\" (UniqueName: \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:08.902545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.902451    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:08.926219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:08.926194    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:09.003916 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.003881    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:09.004047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.003993    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmck6\" (UniqueName: \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:09.006179 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.006150    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:09.025114 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.025093    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmck6\" (UniqueName: \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\") pod \"authorino-68f9555bd6-jmjc6\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") " pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:09.164040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.164002    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:38:09.164040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.164023    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"]
Oct 30 11:38:09.168152 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.168134    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.185439 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.185409    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 11:38:09.185439 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.185422    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5qmrg\""
Oct 30 11:38:09.204060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.199154    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"]
Oct 30 11:38:09.231303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.231277    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"]
Oct 30 11:38:09.236857 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:38:09.236829    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b9e879_cbef_433f_8388_15fc39e8b78c.slice/crio-8013f36586c4a68052785ddee1fb0e512beb0bcf1631784cf44e158e117153de WatchSource:0}: Error finding container 8013f36586c4a68052785ddee1fb0e512beb0bcf1631784cf44e158e117153de: Status 404 returned error can't find the container with id 8013f36586c4a68052785ddee1fb0e512beb0bcf1631784cf44e158e117153de
Oct 30 11:38:09.306130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306094    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxn4c\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-kube-api-access-mxn4c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306145    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306212    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306297    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306357    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306452 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306387    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306484 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306454    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9cb544-80ec-428b-b007-bff354808519-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306484 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306476    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9cb544-80ec-428b-b007-bff354808519-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.306544 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.306505    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407173    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407208    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxn4c\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-kube-api-access-mxn4c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407237    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407253    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407288    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407314    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407348    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407392    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9cb544-80ec-428b-b007-bff354808519-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407757 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407425    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9cb544-80ec-428b-b007-bff354808519-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407811 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407754    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407868 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407808    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.407953 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407915    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.408074 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.407984    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.408131 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.408119    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9cb544-80ec-428b-b007-bff354808519-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.409642 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.409620    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9cb544-80ec-428b-b007-bff354808519-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.409782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.409766    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9cb544-80ec-428b-b007-bff354808519-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.428896 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.428817    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.429015 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.428993    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxn4c\" (UniqueName: \"kubernetes.io/projected/ec9cb544-80ec-428b-b007-bff354808519-kube-api-access-mxn4c\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-z7kbc\" (UID: \"ec9cb544-80ec-428b-b007-bff354808519\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.467511 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.467464    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:38:09.470037 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:38:09.470007    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe713ca6_b0aa_49c8_9496_ab12690e66ea.slice/crio-bd1426adb7af3004a590f0a3cb09cc83a04cdd34ed50af916514624b7ee15006 WatchSource:0}: Error finding container bd1426adb7af3004a590f0a3cb09cc83a04cdd34ed50af916514624b7ee15006: Status 404 returned error can't find the container with id bd1426adb7af3004a590f0a3cb09cc83a04cdd34ed50af916514624b7ee15006
Oct 30 11:38:09.498881 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.498851    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:09.611628 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.611601    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" event={"ID":"fe713ca6-b0aa-49c8-9496-ab12690e66ea","Type":"ContainerStarted","Data":"bd1426adb7af3004a590f0a3cb09cc83a04cdd34ed50af916514624b7ee15006"}
Oct 30 11:38:09.613157 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.613126    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz" event={"ID":"c6b9e879-cbef-433f-8388-15fc39e8b78c","Type":"ContainerStarted","Data":"c7cd423ff6212337a4488c7da3063cefa46531029d1863fa5e9a203c24c8088a"}
Oct 30 11:38:09.613263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.613188    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz" event={"ID":"c6b9e879-cbef-433f-8388-15fc39e8b78c","Type":"ContainerStarted","Data":"8013f36586c4a68052785ddee1fb0e512beb0bcf1631784cf44e158e117153de"}
Oct 30 11:38:09.613263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.613201    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:09.613263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.613236    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-xh6qt"
Oct 30 11:38:09.613525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.613501    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:09.620761 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.620741    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:09.651032 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.650979    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz" podStartSLOduration=1.329808217 podStartE2EDuration="1.650967371s" totalImagesPullingTime="321.159154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:09.650696805 +0000 UTC m=+488.870981715" watchObservedRunningTime="2026-10-30 11:38:09.650967371 +0000 UTC m=+488.871252290"
Oct 30 11:38:09.709060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.709034    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\" (UniqueName: \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\") pod \"879ba9d2-9e13-4282-a477-82fc5286ec60\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") "
Oct 30 11:38:09.709185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.709070    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\" (UniqueName: \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\") pod \"879ba9d2-9e13-4282-a477-82fc5286ec60\" (UID: \"879ba9d2-9e13-4282-a477-82fc5286ec60\") "
Oct 30 11:38:09.711630 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.711608    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf" pod "879ba9d2-9e13-4282-a477-82fc5286ec60" (UID: "879ba9d2-9e13-4282-a477-82fc5286ec60"). InnerVolumeSpecName "kube-api-access-76pxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:38:09.711630 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.711617    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert" pod "879ba9d2-9e13-4282-a477-82fc5286ec60" (UID: "879ba9d2-9e13-4282-a477-82fc5286ec60"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:38:09.748229 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.748158    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xh6qt"]
Oct 30 11:38:09.754058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.754033    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-xh6qt"]
Oct 30 11:38:09.800739 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.800712    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"]
Oct 30 11:38:09.803513 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:38:09.803488    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9cb544_80ec_428b_b007_bff354808519.slice/crio-0f63fa2080325bd8b1ac15b9897f3fba0bf859830235fedaff5fa8e3f5a70470 WatchSource:0}: Error finding container 0f63fa2080325bd8b1ac15b9897f3fba0bf859830235fedaff5fa8e3f5a70470: Status 404 returned error can't find the container with id 0f63fa2080325bd8b1ac15b9897f3fba0bf859830235fedaff5fa8e3f5a70470
Oct 30 11:38:09.810542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.810519    4453 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/879ba9d2-9e13-4282-a477-82fc5286ec60-tls-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:09.810627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:09.810543    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76pxf\" (UniqueName: \"kubernetes.io/projected/879ba9d2-9e13-4282-a477-82fc5286ec60-kube-api-access-76pxf\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:10.619474 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.619435    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" event={"ID":"ec9cb544-80ec-428b-b007-bff354808519","Type":"ContainerStarted","Data":"0f63fa2080325bd8b1ac15b9897f3fba0bf859830235fedaff5fa8e3f5a70470"}
Oct 30 11:38:10.620061 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.619499    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8qghd"
Oct 30 11:38:10.689395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.689357    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8qghd"]
Oct 30 11:38:10.693530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.692963    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8qghd"]
Oct 30 11:38:10.910285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.910210    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d63407-3c94-4667-ac36-ea1d26bb3775" path="/var/lib/kubelet/pods/57d63407-3c94-4667-ac36-ea1d26bb3775/volumes"
Oct 30 11:38:10.910724 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:10.910706    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879ba9d2-9e13-4282-a477-82fc5286ec60" path="/var/lib/kubelet/pods/879ba9d2-9e13-4282-a477-82fc5286ec60/volumes"
Oct 30 11:38:12.374297 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:12.374264    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:38:12.374627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:12.374346    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:38:12.374627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:12.374387    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:38:12.630782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:12.630717    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" event={"ID":"ec9cb544-80ec-428b-b007-bff354808519","Type":"ContainerStarted","Data":"8704f46a0a7e93c33b6598448ab6bda37b547ace5a1d5d5a4f9aed4150eafcc7"}
Oct 30 11:38:12.688736 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:12.688675    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podStartSLOduration=1.120812678 podStartE2EDuration="3.688654534s" totalImagesPullingTime="2.567841856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:12.68053663 +0000 UTC m=+491.900821541" watchObservedRunningTime="2026-10-30 11:38:12.688654534 +0000 UTC m=+491.908939445"
Oct 30 11:38:13.507232 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:13.507197    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:13.507979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:13.507950    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 11:38:13.508161 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:13.508130    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podUID="ec9cb544-80ec-428b-b007-bff354808519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 11:38:13.636148 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:13.636116    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" event={"ID":"fe713ca6-b0aa-49c8-9496-ab12690e66ea","Type":"ContainerStarted","Data":"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"}
Oct 30 11:38:13.672683 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:13.672611    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" podStartSLOduration=2.482670666 podStartE2EDuration="5.67259781s" totalImagesPullingTime="3.189927144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:38:13.67213208 +0000 UTC m=+492.892416989" watchObservedRunningTime="2026-10-30 11:38:13.67259781 +0000 UTC m=+492.892882720"
Oct 30 11:38:14.499746 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:14.499710    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 11:38:14.500064 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:14.500034    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podUID="ec9cb544-80ec-428b-b007-bff354808519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 11:38:15.506812 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.506770    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 11:38:15.507432 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.507398    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podUID="ec9cb544-80ec-428b-b007-bff354808519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 11:38:15.626303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.626271    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c6d5fb647-8rlbz"
Oct 30 11:38:15.707352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.703242    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:38:15.707352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.703488    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" podUID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" containerName="maas-api" containerID="cri-o://69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6" gracePeriod=30
Oct 30 11:38:15.965657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:15.965625    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:38:16.067030 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.066998    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\" (UniqueName: \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\") pod \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") "
Oct 30 11:38:16.067187 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.067038    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\") pod \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") "
Oct 30 11:38:16.067187 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.067057    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\") pod \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") "
Oct 30 11:38:16.067187 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.067100    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\") pod \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\" (UID: \"9912f4e3-2e4b-4c7e-a04c-24260e5c301f\") "
Oct 30 11:38:16.067589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.067556    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca" pod "9912f4e3-2e4b-4c7e-a04c-24260e5c301f" (UID: "9912f4e3-2e4b-4c7e-a04c-24260e5c301f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:38:16.069451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.069421    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p" pod "9912f4e3-2e4b-4c7e-a04c-24260e5c301f" (UID: "9912f4e3-2e4b-4c7e-a04c-24260e5c301f"). InnerVolumeSpecName "kube-api-access-vxx7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:38:16.069693 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.069671    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls" pod "9912f4e3-2e4b-4c7e-a04c-24260e5c301f" (UID: "9912f4e3-2e4b-4c7e-a04c-24260e5c301f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:38:16.069693 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.069687    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs" pod "9912f4e3-2e4b-4c7e-a04c-24260e5c301f" (UID: "9912f4e3-2e4b-4c7e-a04c-24260e5c301f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:38:16.168465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.168431    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxx7p\" (UniqueName: \"kubernetes.io/projected/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-kube-api-access-vxx7p\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:16.168465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.168459    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:16.168465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.168468    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:16.168691 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.168476    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9912f4e3-2e4b-4c7e-a04c-24260e5c301f-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:38:16.509406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.506707    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 11:38:16.509406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.506768    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podUID="ec9cb544-80ec-428b-b007-bff354808519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 11:38:16.649091 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.649051    4453 generic.go:356] "Generic (PLEG): container finished" podID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" containerID="69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6" exitCode=0
Oct 30 11:38:16.649258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.649105    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"
Oct 30 11:38:16.649258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.649138    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" event={"ID":"9912f4e3-2e4b-4c7e-a04c-24260e5c301f","Type":"ContainerDied","Data":"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"}
Oct 30 11:38:16.649258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.649198    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t" event={"ID":"9912f4e3-2e4b-4c7e-a04c-24260e5c301f","Type":"ContainerDied","Data":"efaa5def97fb61dd13b2a503472abeed9509ed7d78c03dcb01897a017994b6e1"}
Oct 30 11:38:16.649258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.649226    4453 scope.go:122] "RemoveContainer" containerID="69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"
Oct 30 11:38:16.657708 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.657692    4453 scope.go:122] "RemoveContainer" containerID="69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"
Oct 30 11:38:16.657977 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:38:16.657957    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6\": container with ID starting with 69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6 not found: ID does not exist" containerID="69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"
Oct 30 11:38:16.658025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.657986    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6"} err="failed to get container status \"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6\": rpc error: code = NotFound desc = could not find container \"69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6\": container with ID starting with 69d36865152e34907f22ab32fc336bb92c3c02cbdebdd0529f2eb392ba169ff6 not found: ID does not exist"
Oct 30 11:38:16.715195 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.711400    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:38:16.720489 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.720464    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c59648cf5-vjj4t"]
Oct 30 11:38:16.916588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:16.916558    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" path="/var/lib/kubelet/pods/9912f4e3-2e4b-4c7e-a04c-24260e5c301f/volumes"
Oct 30 11:38:17.511355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:17.510750    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 11:38:17.511355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:17.510824    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc" podUID="ec9cb544-80ec-428b-b007-bff354808519" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 11:38:18.503835 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:18.503800    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:18.504080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:18.504059    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:18.504845 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:38:18.504825    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-z7kbc"
Oct 30 11:38:37.569016 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:38:37.568975    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:39:07.569007 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:39:07.568960    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:39:13.672440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.672399    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:13.672947 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.672925    4453 cpu_manager.go:414] "containerMap: removing container" podUID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" containerName="maas-api"
Oct 30 11:39:13.673024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.672949    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" containerName="maas-api"
Oct 30 11:39:13.673082 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.673071    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="9912f4e3-2e4b-4c7e-a04c-24260e5c301f" containerName="maas-api"
Oct 30 11:39:13.676893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.676871    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.689425 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.689406    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 11:39:13.699057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.699036    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-r6wwd\""
Oct 30 11:39:13.699057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.699036    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 11:39:13.700308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.700292    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.700382 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.700367    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68pt\" (UniqueName: \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.706469 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.706451    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:13.800701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.800672    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.800851 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.800714    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68pt\" (UniqueName: \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.803076 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.803051    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.821962 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.821933    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68pt\" (UniqueName: \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\") pod \"ai-gateway-controller-7d85675f98-6gvc9\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") " pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:13.989959 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:13.989881    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:14.279674 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:14.279542    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:14.282577 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:14.282540    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1e6401_8620_48c7_9cfc_991fdbb3a294.slice/crio-9d91b78ee0f3ee1a41a3b3320c82538d117cbe3f32436d4d22a2cab86883fd8b WatchSource:0}: Error finding container 9d91b78ee0f3ee1a41a3b3320c82538d117cbe3f32436d4d22a2cab86883fd8b: Status 404 returned error can't find the container with id 9d91b78ee0f3ee1a41a3b3320c82538d117cbe3f32436d4d22a2cab86883fd8b
Oct 30 11:39:14.861544 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:14.861512    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" event={"ID":"1a1e6401-8620-48c7-9cfc-991fdbb3a294","Type":"ContainerStarted","Data":"9d91b78ee0f3ee1a41a3b3320c82538d117cbe3f32436d4d22a2cab86883fd8b"}
Oct 30 11:39:16.870876 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:16.870836    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" event={"ID":"1a1e6401-8620-48c7-9cfc-991fdbb3a294","Type":"ContainerStarted","Data":"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"}
Oct 30 11:39:16.871249 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:16.870907    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:16.906764 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:16.906712    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" podStartSLOduration=2.148946276 podStartE2EDuration="3.906694133s" totalImagesPullingTime="1.757747857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:16.905902475 +0000 UTC m=+556.126187384" watchObservedRunningTime="2026-10-30 11:39:16.906694133 +0000 UTC m=+556.126979048"
Oct 30 11:39:17.335814 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.335731    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"]
Oct 30 11:39:17.339150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.339130    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.353734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.353711    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 11:39:17.353850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.353798    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cdzbd\""
Oct 30 11:39:17.361972 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.361945    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"]
Oct 30 11:39:17.427516 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.427471    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmwp\" (UniqueName: \"kubernetes.io/projected/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-kube-api-access-4kmwp\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.427681 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.427563    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.436838 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.436796    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bqrn5"]
Oct 30 11:39:17.440560 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.440536    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.469419 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.469377    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bqrn5"]
Oct 30 11:39:17.528401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.528353    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.528401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.528407    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6x6\" (UniqueName: \"kubernetes.io/projected/30d55454-08b7-4aca-ad04-f26a881da3d2-kube-api-access-mj6x6\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.528661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.528459    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d55454-08b7-4aca-ad04-f26a881da3d2-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.528661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.528564    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmwp\" (UniqueName: \"kubernetes.io/projected/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-kube-api-access-4kmwp\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.529150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.529126    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.549671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.549635    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmwp\" (UniqueName: \"kubernetes.io/projected/2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674-kube-api-access-4kmwp\") pod \"payload-pre-processing-7b98bb678c-kfvvk\" (UID: \"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.629584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.629544    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d55454-08b7-4aca-ad04-f26a881da3d2-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.629766 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.629639    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6x6\" (UniqueName: \"kubernetes.io/projected/30d55454-08b7-4aca-ad04-f26a881da3d2-kube-api-access-mj6x6\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.630252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.630230    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d55454-08b7-4aca-ad04-f26a881da3d2-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.650737 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.650712    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6x6\" (UniqueName: \"kubernetes.io/projected/30d55454-08b7-4aca-ad04-f26a881da3d2-kube-api-access-mj6x6\") pod \"payload-processing-5f78687d5b-bqrn5\" (UID: \"30d55454-08b7-4aca-ad04-f26a881da3d2\") " pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.650854 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.650846    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:17.753617 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.753584    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:17.942991 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:17.942969    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"]
Oct 30 11:39:17.945898 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:17.945864    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d18c6a5_9f7b_4051_9ffb_cdfb6f93a674.slice/crio-cf18ebd9ec9c73a9c4da90d04ffdf9ec082b73b4d9645133fa8692750c343933 WatchSource:0}: Error finding container cf18ebd9ec9c73a9c4da90d04ffdf9ec082b73b4d9645133fa8692750c343933: Status 404 returned error can't find the container with id cf18ebd9ec9c73a9c4da90d04ffdf9ec082b73b4d9645133fa8692750c343933
Oct 30 11:39:18.046944 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:18.046919    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bqrn5"]
Oct 30 11:39:18.049158 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:18.049128    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d55454_08b7_4aca_ad04_f26a881da3d2.slice/crio-b0dcb746d67e9956347e4931820b1fc67bcf562ef02eb19e16352e11eab60a37 WatchSource:0}: Error finding container b0dcb746d67e9956347e4931820b1fc67bcf562ef02eb19e16352e11eab60a37: Status 404 returned error can't find the container with id b0dcb746d67e9956347e4931820b1fc67bcf562ef02eb19e16352e11eab60a37
Oct 30 11:39:18.882672 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:18.882591    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5" event={"ID":"30d55454-08b7-4aca-ad04-f26a881da3d2","Type":"ContainerStarted","Data":"b0dcb746d67e9956347e4931820b1fc67bcf562ef02eb19e16352e11eab60a37"}
Oct 30 11:39:18.884149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:18.884034    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk" event={"ID":"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674","Type":"ContainerStarted","Data":"cf18ebd9ec9c73a9c4da90d04ffdf9ec082b73b4d9645133fa8692750c343933"}
Oct 30 11:39:20.896264 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.896229    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5" event={"ID":"30d55454-08b7-4aca-ad04-f26a881da3d2","Type":"ContainerStarted","Data":"f90509a3eeecebbe2136313c2b814e913381d2d9fcd2ff682d6cdb7435d4e6ed"}
Oct 30 11:39:20.896769 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.896356    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:20.897779 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.897756    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk" event={"ID":"2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674","Type":"ContainerStarted","Data":"5d886bd67435e8371e9f42b5f121a12f395056cd86bf52b7422e2cb0e61df859"}
Oct 30 11:39:20.897890 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.897873    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:20.931929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.931885    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5" podStartSLOduration=1.6739444159999999 podStartE2EDuration="3.931873511s" totalImagesPullingTime="2.257929095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:20.931380939 +0000 UTC m=+560.151665849" watchObservedRunningTime="2026-10-30 11:39:20.931873511 +0000 UTC m=+560.152158421"
Oct 30 11:39:20.968178 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:20.968138    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk" podStartSLOduration=1.661062085 podStartE2EDuration="3.968125461s" totalImagesPullingTime="2.307063376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:20.966759201 +0000 UTC m=+560.187044123" watchObservedRunningTime="2026-10-30 11:39:20.968125461 +0000 UTC m=+560.188410371"
Oct 30 11:39:27.877117 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:27.877085    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:28.168211 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.168136    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6558dd845f-gqggd"]
Oct 30 11:39:28.171678 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.171657    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.192693 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.192668    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6558dd845f-gqggd"]
Oct 30 11:39:28.210547 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.210523    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6c2\" (UniqueName: \"kubernetes.io/projected/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-kube-api-access-ws6c2\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.210679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.210605    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-metrics-certs\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.311581 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.311554    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6c2\" (UniqueName: \"kubernetes.io/projected/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-kube-api-access-ws6c2\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.311697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.311604    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-metrics-certs\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.313877 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.313858    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-metrics-certs\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.332393 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.332370    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6c2\" (UniqueName: \"kubernetes.io/projected/5bea9e70-e94d-43a7-8f3f-8be36598f8fc-kube-api-access-ws6c2\") pod \"ai-gateway-controller-6558dd845f-gqggd\" (UID: \"5bea9e70-e94d-43a7-8f3f-8be36598f8fc\") " pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.483568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.483492    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:28.774145 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.774077    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6558dd845f-gqggd"]
Oct 30 11:39:28.777434 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:28.777397    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bea9e70_e94d_43a7_8f3f_8be36598f8fc.slice/crio-04453ea3fa5bfb05e32e68e0519019c91785016f973b4d9603580a0a7da27fc9 WatchSource:0}: Error finding container 04453ea3fa5bfb05e32e68e0519019c91785016f973b4d9603580a0a7da27fc9: Status 404 returned error can't find the container with id 04453ea3fa5bfb05e32e68e0519019c91785016f973b4d9603580a0a7da27fc9
Oct 30 11:39:28.925055 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:28.925026    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd" event={"ID":"5bea9e70-e94d-43a7-8f3f-8be36598f8fc","Type":"ContainerStarted","Data":"04453ea3fa5bfb05e32e68e0519019c91785016f973b4d9603580a0a7da27fc9"}
Oct 30 11:39:29.930528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:29.930489    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd" event={"ID":"5bea9e70-e94d-43a7-8f3f-8be36598f8fc","Type":"ContainerStarted","Data":"5e27db98ede6711a13c93d4f353202d0093d566143df079846b4e159803f412e"}
Oct 30 11:39:29.931093 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:29.930575    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:29.966395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:29.966319    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd" podStartSLOduration=1.603597743 podStartE2EDuration="1.966305366s" totalImagesPullingTime="362.707623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:29.965915842 +0000 UTC m=+569.186200745" watchObservedRunningTime="2026-10-30 11:39:29.966305366 +0000 UTC m=+569.186590299"
Oct 30 11:39:31.904969 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:31.904934    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-kfvvk"
Oct 30 11:39:31.905412 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:31.905012    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-bqrn5"
Oct 30 11:39:37.568896 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:39:37.568859    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:39:40.937527 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:40.937491    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6558dd845f-gqggd"
Oct 30 11:39:41.006195 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.006159    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:41.006418 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.006370    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" podUID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" containerName="manager" containerID="cri-o://23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba" gracePeriod=30
Oct 30 11:39:41.251551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.251527    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:41.323897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.323873    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\") pod \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") "
Oct 30 11:39:41.324037 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.323915    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\" (UniqueName: \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\") pod \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\" (UID: \"1a1e6401-8620-48c7-9cfc-991fdbb3a294\") "
Oct 30 11:39:41.326017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.325985    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt" pod "1a1e6401-8620-48c7-9cfc-991fdbb3a294" (UID: "1a1e6401-8620-48c7-9cfc-991fdbb3a294"). InnerVolumeSpecName "kube-api-access-x68pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:39:41.326017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.326004    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs" pod "1a1e6401-8620-48c7-9cfc-991fdbb3a294" (UID: "1a1e6401-8620-48c7-9cfc-991fdbb3a294"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:39:41.424924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.424897    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a1e6401-8620-48c7-9cfc-991fdbb3a294-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:39:41.424924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.424922    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68pt\" (UniqueName: \"kubernetes.io/projected/1a1e6401-8620-48c7-9cfc-991fdbb3a294-kube-api-access-x68pt\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:39:41.983558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.983526    4453 generic.go:356] "Generic (PLEG): container finished" podID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" containerID="23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba" exitCode=0
Oct 30 11:39:41.984019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.983577    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9"
Oct 30 11:39:41.984019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.983614    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" event={"ID":"1a1e6401-8620-48c7-9cfc-991fdbb3a294","Type":"ContainerDied","Data":"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"}
Oct 30 11:39:41.984019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.983715    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-6gvc9" event={"ID":"1a1e6401-8620-48c7-9cfc-991fdbb3a294","Type":"ContainerDied","Data":"9d91b78ee0f3ee1a41a3b3320c82538d117cbe3f32436d4d22a2cab86883fd8b"}
Oct 30 11:39:41.984019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.983746    4453 scope.go:122] "RemoveContainer" containerID="23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"
Oct 30 11:39:41.992218 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.992203    4453 scope.go:122] "RemoveContainer" containerID="23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"
Oct 30 11:39:41.992487 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:39:41.992469    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba\": container with ID starting with 23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba not found: ID does not exist" containerID="23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"
Oct 30 11:39:41.992538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:41.992493    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba"} err="failed to get container status \"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba\": rpc error: code = NotFound desc = could not find container \"23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba\": container with ID starting with 23b4dfc50bcfb605fff040a10136b11f50212f79cba047517acd9eedeffa88ba not found: ID does not exist"
Oct 30 11:39:42.031374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:42.031354    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:42.040770 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:42.040743    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-6gvc9"]
Oct 30 11:39:42.909433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:42.909399    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" path="/var/lib/kubelet/pods/1a1e6401-8620-48c7-9cfc-991fdbb3a294/volumes"
Oct 30 11:39:48.783032 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.782955    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"]
Oct 30 11:39:48.783517 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.783481    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" containerName="manager"
Oct 30 11:39:48.783517 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.783501    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" containerName="manager"
Oct 30 11:39:48.783982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.783959    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1e6401-8620-48c7-9cfc-991fdbb3a294" containerName="manager"
Oct 30 11:39:48.787836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.787812    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.801596 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.801574    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 11:39:48.810912 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.810890    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 11:39:48.810912 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.810904    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 11:39:48.810912 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.810896    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gkjwq\""
Oct 30 11:39:48.812499 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.812474    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"]
Oct 30 11:39:48.895106 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895064    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdmm\" (UniqueName: \"kubernetes.io/projected/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-kube-api-access-pqdmm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.895106 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895107    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.895403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895145    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.895403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895265    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.895403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895383    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.895527 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.895425    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996693    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996731    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996755    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996795    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996832    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdmm\" (UniqueName: \"kubernetes.io/projected/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-kube-api-access-pqdmm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.996979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.996964    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.997222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.997199    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.997342 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.997310    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.997397 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.997377    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.999127 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.999106    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:48.999283 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:48.999264    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:49.017776 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:49.017746    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdmm\" (UniqueName: \"kubernetes.io/projected/7d6a9d3d-d4f6-4bd6-acab-c305192ca954-kube-api-access-pqdmm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb\" (UID: \"7d6a9d3d-d4f6-4bd6-acab-c305192ca954\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:49.103114 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:49.103050    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:49.399414 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:49.399393    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"]
Oct 30 11:39:49.402043 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:49.402017    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6a9d3d_d4f6_4bd6_acab_c305192ca954.slice/crio-ad3d2996950018cdfcaf1a15bfe0180970c2ec858645fdbd993f8cb7c0efb182 WatchSource:0}: Error finding container ad3d2996950018cdfcaf1a15bfe0180970c2ec858645fdbd993f8cb7c0efb182: Status 404 returned error can't find the container with id ad3d2996950018cdfcaf1a15bfe0180970c2ec858645fdbd993f8cb7c0efb182
Oct 30 11:39:50.021444 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:50.021399    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb" event={"ID":"7d6a9d3d-d4f6-4bd6-acab-c305192ca954","Type":"ContainerStarted","Data":"ad3d2996950018cdfcaf1a15bfe0180970c2ec858645fdbd993f8cb7c0efb182"}
Oct 30 11:39:51.027504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.027416    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb" event={"ID":"7d6a9d3d-d4f6-4bd6-acab-c305192ca954","Type":"ContainerStarted","Data":"061ed3c767dcff4dee8c56d5458eef8d98a40878aae64721ffaeab3f1901e9ef"}
Oct 30 11:39:51.064388 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.064316    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb" podStartSLOduration=1.797222959 podStartE2EDuration="3.064301147s" totalImagesPullingTime="1.267078188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:51.063699944 +0000 UTC m=+590.283984853" watchObservedRunningTime="2026-10-30 11:39:51.064301147 +0000 UTC m=+590.284586060"
Oct 30 11:39:51.103250 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.103203    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:51.894914 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.894882    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"]
Oct 30 11:39:51.898091 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.898066    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:51.919479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.919454    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 11:39:51.921039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:51.921013    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"]
Oct 30 11:39:52.020948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.020916    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fab727a-0746-46f5-8bf8-d230acc8ee08-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.020948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.020952    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.021192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.020973    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.021192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.021027    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.021192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.021121    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.021192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.021157    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5m6r\" (UniqueName: \"kubernetes.io/projected/3fab727a-0746-46f5-8bf8-d230acc8ee08-kube-api-access-b5m6r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122314 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fab727a-0746-46f5-8bf8-d230acc8ee08-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122314 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122316    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122348    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122363    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122389    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122414    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5m6r\" (UniqueName: \"kubernetes.io/projected/3fab727a-0746-46f5-8bf8-d230acc8ee08-kube-api-access-b5m6r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122787    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.122867 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122823    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.123088 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.122872    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.124655 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.124637    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fab727a-0746-46f5-8bf8-d230acc8ee08-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.124823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.124806    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fab727a-0746-46f5-8bf8-d230acc8ee08-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.143313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.143287    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5m6r\" (UniqueName: \"kubernetes.io/projected/3fab727a-0746-46f5-8bf8-d230acc8ee08-kube-api-access-b5m6r\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kzmlb\" (UID: \"3fab727a-0746-46f5-8bf8-d230acc8ee08\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.218048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.217797    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:52.517113 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:52.517043    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"]
Oct 30 11:39:52.520217 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:52.520187    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fab727a_0746_46f5_8bf8_d230acc8ee08.slice/crio-421c258f1601bc4df9aae2a89c856c43174123cee0241c555d68a9897ddf0b4f WatchSource:0}: Error finding container 421c258f1601bc4df9aae2a89c856c43174123cee0241c555d68a9897ddf0b4f: Status 404 returned error can't find the container with id 421c258f1601bc4df9aae2a89c856c43174123cee0241c555d68a9897ddf0b4f
Oct 30 11:39:53.037855 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.037805    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb" event={"ID":"3fab727a-0746-46f5-8bf8-d230acc8ee08","Type":"ContainerStarted","Data":"624eb7f5c2d91bec1dd70a1d6a6f13773be28b65aec45b614f5d3fff5173c2f9"}
Oct 30 11:39:53.038041 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.037883    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb" event={"ID":"3fab727a-0746-46f5-8bf8-d230acc8ee08","Type":"ContainerStarted","Data":"421c258f1601bc4df9aae2a89c856c43174123cee0241c555d68a9897ddf0b4f"}
Oct 30 11:39:53.067860 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.067831    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"]
Oct 30 11:39:53.075304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.075278    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.076989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.076667    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb" podStartSLOduration=1.900429787 podStartE2EDuration="2.076652461s" totalImagesPullingTime="176.222674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:53.074836882 +0000 UTC m=+592.295121792" watchObservedRunningTime="2026-10-30 11:39:53.076652461 +0000 UTC m=+592.296937374"
Oct 30 11:39:53.087681 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.087654    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 11:39:53.100004 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.099978    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"]
Oct 30 11:39:53.218469 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.218434    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:39:53.231043 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231009    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2bl\" (UniqueName: \"kubernetes.io/projected/cd9e6fb6-50e7-43b1-b99b-51bf284196df-kube-api-access-fd2bl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.231209 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231060    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.231209 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231176    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.231375 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231249    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.231375 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231281    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.231504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.231476    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332558    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332637    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2bl\" (UniqueName: \"kubernetes.io/projected/cd9e6fb6-50e7-43b1-b99b-51bf284196df-kube-api-access-fd2bl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332664    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332708    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332740    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.332834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.332766    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.333068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.333046    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.333108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.333100    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.333206 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.333184    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.334942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.334916    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd9e6fb6-50e7-43b1-b99b-51bf284196df-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.335193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.335176    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd9e6fb6-50e7-43b1-b99b-51bf284196df-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.354039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.354010    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2bl\" (UniqueName: \"kubernetes.io/projected/cd9e6fb6-50e7-43b1-b99b-51bf284196df-kube-api-access-fd2bl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-f9zsw\" (UID: \"cd9e6fb6-50e7-43b1-b99b-51bf284196df\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.390793 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.390761    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:53.687224 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:53.687192    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"]
Oct 30 11:39:53.689777 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:53.689748    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9e6fb6_50e7_43b1_b99b_51bf284196df.slice/crio-f9400ce2c4f744c54bc40748033a094ef34b944f9d83d7f72550c2d5a8065102 WatchSource:0}: Error finding container f9400ce2c4f744c54bc40748033a094ef34b944f9d83d7f72550c2d5a8065102: Status 404 returned error can't find the container with id f9400ce2c4f744c54bc40748033a094ef34b944f9d83d7f72550c2d5a8065102
Oct 30 11:39:54.045274 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:54.045240    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw" event={"ID":"cd9e6fb6-50e7-43b1-b99b-51bf284196df","Type":"ContainerStarted","Data":"22f26bbd4435acc7a704f8efc76ab51f270bed4e98102841366f65dc7c5858e7"}
Oct 30 11:39:54.045464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:54.045298    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw" event={"ID":"cd9e6fb6-50e7-43b1-b99b-51bf284196df","Type":"ContainerStarted","Data":"f9400ce2c4f744c54bc40748033a094ef34b944f9d83d7f72550c2d5a8065102"}
Oct 30 11:39:54.085275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:54.085220    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw" podStartSLOduration=0.905445888 podStartE2EDuration="1.085206096s" totalImagesPullingTime="179.760208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:54.084771351 +0000 UTC m=+593.305056262" watchObservedRunningTime="2026-10-30 11:39:54.085206096 +0000 UTC m=+593.305491069"
Oct 30 11:39:54.391786 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:54.391752    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:39:55.023855 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.023818    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"]
Oct 30 11:39:55.028042 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.028012    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.052276 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.052246    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 11:39:55.059366 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.055378    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"]
Oct 30 11:39:55.149603 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149570    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9363a866-fbd8-469d-939c-44211df30df7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.149793 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149628    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.149793 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149711    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.149793 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149754    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f6b\" (UniqueName: \"kubernetes.io/projected/9363a866-fbd8-469d-939c-44211df30df7-kube-api-access-v7f6b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.149973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149892    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.149973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.149952    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.250976 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.250899    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251217 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251075    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9363a866-fbd8-469d-939c-44211df30df7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251217 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251157    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251217 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251206    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251252    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f6b\" (UniqueName: \"kubernetes.io/projected/9363a866-fbd8-469d-939c-44211df30df7-kube-api-access-v7f6b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251361    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251388    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251725 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251571    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.251822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.251799    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.253651 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.253624    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9363a866-fbd8-469d-939c-44211df30df7-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.253779 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.253758    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9363a866-fbd8-469d-939c-44211df30df7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.272455 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.272420    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f6b\" (UniqueName: \"kubernetes.io/projected/9363a866-fbd8-469d-939c-44211df30df7-kube-api-access-v7f6b\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-mz9p2\" (UID: \"9363a866-fbd8-469d-939c-44211df30df7\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.342530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.342442    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:55.642774 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:55.642730    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"]
Oct 30 11:39:55.645319 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:39:55.645291    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9363a866_fbd8_469d_939c_44211df30df7.slice/crio-43d04b4d6760d0f6e470bc918d9a7fda2e60f85175e1d7e721b012bf7a5309f8 WatchSource:0}: Error finding container 43d04b4d6760d0f6e470bc918d9a7fda2e60f85175e1d7e721b012bf7a5309f8: Status 404 returned error can't find the container with id 43d04b4d6760d0f6e470bc918d9a7fda2e60f85175e1d7e721b012bf7a5309f8
Oct 30 11:39:56.056022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:56.055978    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2" event={"ID":"9363a866-fbd8-469d-939c-44211df30df7","Type":"ContainerStarted","Data":"e2d606a2f532704cc36902cc7ec0982552b5e4a0b40e3249fdf02a42e198bd91"}
Oct 30 11:39:56.056198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:56.056078    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2" event={"ID":"9363a866-fbd8-469d-939c-44211df30df7","Type":"ContainerStarted","Data":"43d04b4d6760d0f6e470bc918d9a7fda2e60f85175e1d7e721b012bf7a5309f8"}
Oct 30 11:39:56.094778 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:56.094725    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2" podStartSLOduration=1.908436786 podStartE2EDuration="2.09471182s" totalImagesPullingTime="186.275034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:39:56.093303746 +0000 UTC m=+595.313588659" watchObservedRunningTime="2026-10-30 11:39:56.09471182 +0000 UTC m=+595.314996730"
Oct 30 11:39:56.343039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:56.342946    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:39:59.103422 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:59.103382    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:39:59.117969 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:39:59.117940    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:40:00.084279 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.084246    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb"
Oct 30 11:40:00.427903 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.427867    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"]
Oct 30 11:40:00.432414 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.432385    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.455961 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.455930    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"]
Oct 30 11:40:00.517757 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.517718    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.517926 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.517789    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.517926 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.517869    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvh8l\" (UniqueName: \"kubernetes.io/projected/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-kube-api-access-mvh8l\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.517926 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.517900    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.518049 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.517933    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.518049 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.518011    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.537777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.537732    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 11:40:00.619409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619367    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619437    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619469    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619549    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619618    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619649    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh8l\" (UniqueName: \"kubernetes.io/projected/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-kube-api-access-mvh8l\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.619991 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.619949    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.620044 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.620017    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.620198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.620179    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.622281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.622257    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.622556 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.622540    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.642955 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.642930    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvh8l\" (UniqueName: \"kubernetes.io/projected/fd3d1ac0-2e49-4c83-9c5b-f09a45326725-kube-api-access-mvh8l\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-99czt\" (UID: \"fd3d1ac0-2e49-4c83-9c5b-f09a45326725\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:00.749735 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:00.749646    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:01.049520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:01.049441    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"]
Oct 30 11:40:01.051547 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:40:01.051520    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3d1ac0_2e49_4c83_9c5b_f09a45326725.slice/crio-5bfe04733e7d18650d1577fdaafb9ae3f483d3f066d5e8c7a3ae7844d9393e4e WatchSource:0}: Error finding container 5bfe04733e7d18650d1577fdaafb9ae3f483d3f066d5e8c7a3ae7844d9393e4e: Status 404 returned error can't find the container with id 5bfe04733e7d18650d1577fdaafb9ae3f483d3f066d5e8c7a3ae7844d9393e4e
Oct 30 11:40:01.078267 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:01.078234    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt" event={"ID":"fd3d1ac0-2e49-4c83-9c5b-f09a45326725","Type":"ContainerStarted","Data":"5bfe04733e7d18650d1577fdaafb9ae3f483d3f066d5e8c7a3ae7844d9393e4e"}
Oct 30 11:40:02.094370 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.094303    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt" event={"ID":"fd3d1ac0-2e49-4c83-9c5b-f09a45326725","Type":"ContainerStarted","Data":"d3d575f4ef65d9dadd3e737cfac41594558681c363082b991bcdad0cf3af9839"}
Oct 30 11:40:02.136642 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.136582    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt" podStartSLOduration=1.921125205 podStartE2EDuration="2.136564519s" totalImagesPullingTime="215.439314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:02.136051659 +0000 UTC m=+601.356336569" watchObservedRunningTime="2026-10-30 11:40:02.136564519 +0000 UTC m=+601.356849430"
Oct 30 11:40:02.218657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.218622    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:40:02.231771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.231741    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:40:02.436721 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.436690    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"]
Oct 30 11:40:02.440632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.440515    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.457238 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.457207    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 11:40:02.465930 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.465897    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"]
Oct 30 11:40:02.535339 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535301    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.535530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535436    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.535530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535483    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.535648 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535580    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843c47a-0b1d-4004-a852-923b462f0c5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.535702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535639    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.535788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.535743    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ndg\" (UniqueName: \"kubernetes.io/projected/7843c47a-0b1d-4004-a852-923b462f0c5c-kube-api-access-b6ndg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636339    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636391    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636408    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636438    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843c47a-0b1d-4004-a852-923b462f0c5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636484    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636515    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ndg\" (UniqueName: \"kubernetes.io/projected/7843c47a-0b1d-4004-a852-923b462f0c5c-kube-api-access-b6ndg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.636947 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636921    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.637019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.636948    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.637147 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.637126    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.639238 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.639216    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7843c47a-0b1d-4004-a852-923b462f0c5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.639621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.639597    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7843c47a-0b1d-4004-a852-923b462f0c5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.659001 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.658970    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ndg\" (UniqueName: \"kubernetes.io/projected/7843c47a-0b1d-4004-a852-923b462f0c5c-kube-api-access-b6ndg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f\" (UID: \"7843c47a-0b1d-4004-a852-923b462f0c5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:02.750016 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.749941    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:02.753313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:02.753284    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:03.066012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:03.065978    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"]
Oct 30 11:40:03.067367 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:40:03.067322    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7843c47a_0b1d_4004_a852_923b462f0c5c.slice/crio-49bb32ed082b521157d3a0ac015541d67745e56c34d9ffb0f5a1d142e6fe2705 WatchSource:0}: Error finding container 49bb32ed082b521157d3a0ac015541d67745e56c34d9ffb0f5a1d142e6fe2705: Status 404 returned error can't find the container with id 49bb32ed082b521157d3a0ac015541d67745e56c34d9ffb0f5a1d142e6fe2705
Oct 30 11:40:03.102868 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:03.102827    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" event={"ID":"7843c47a-0b1d-4004-a852-923b462f0c5c","Type":"ContainerStarted","Data":"49bb32ed082b521157d3a0ac015541d67745e56c34d9ffb0f5a1d142e6fe2705"}
Oct 30 11:40:03.121192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:03.121150    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kzmlb"
Oct 30 11:40:03.393939 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:03.391710    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:40:03.440369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:03.430537    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:40:04.110767 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:04.110727    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" event={"ID":"7843c47a-0b1d-4004-a852-923b462f0c5c","Type":"ContainerStarted","Data":"b297f616c135b1e7fc1748f6a2d8a1ae4747a46e705f255c38797cfb3a81e614"}
Oct 30 11:40:04.122775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:04.122750    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-f9zsw"
Oct 30 11:40:04.155712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:04.155665    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" podStartSLOduration=1.961404861 podStartE2EDuration="2.155651472s" totalImagesPullingTime="194.246611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:04.154559301 +0000 UTC m=+603.374844224" watchObservedRunningTime="2026-10-30 11:40:04.155651472 +0000 UTC m=+603.375936375"
Oct 30 11:40:04.753882 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:04.753846    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:05.342564 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:05.342524    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:40:05.355227 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:05.355201    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:40:06.135418 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:06.135389    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-mz9p2"
Oct 30 11:40:07.569487 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:40:07.569445    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:40:09.514119 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.514084    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:40:09.518096 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.518075    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.554490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.554464    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 11:40:09.557955 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.557931    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.562223 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.562198    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:40:09.586907 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.586884    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 11:40:09.611118 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611089    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611118 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611121    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611291 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611137    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611291 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611216    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611291 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611254    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611448 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611311    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611448 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611362    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblf2\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611448 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611404    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611448 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611435    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611484    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611515    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611571    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611763 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611658    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611763 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611710    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611763 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611747    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvdw\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611895 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611779    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.611895 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611802    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.611972 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.611904    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712661    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.712698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712699    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712721    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712744    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblf2\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712796    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712824    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712851    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712875    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.712919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712903    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712935    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712966    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.712991    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvdw\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713040    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713082    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713122    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713130    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713187    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713218    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713257    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713286    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.713784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713604    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.714072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713768    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.714072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713814    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.714072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713932    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.714072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.713978    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.714297 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.714201    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.714297 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.714266    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.714830 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.714803    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.715656 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.715632    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.715773 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.715674    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.715945 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.715926    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.716703 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.716672    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.734427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.734401    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.734544 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.734489    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.734604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.734577    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvdw\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-rkv2s\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.735088 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.735072    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblf2\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-fqn85\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:09.832524 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.832447    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:09.871306 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:09.871207    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:10.152794 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.152761    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:40:10.166511 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:40:10.166480    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8718780e_49dd_4e5c_be31_527d4d6590df.slice/crio-67d5a4f6eb0cac72a5932b94fc5f4ff813a042d65e57ba7e0ccde44f40b78834 WatchSource:0}: Error finding container 67d5a4f6eb0cac72a5932b94fc5f4ff813a042d65e57ba7e0ccde44f40b78834: Status 404 returned error can't find the container with id 67d5a4f6eb0cac72a5932b94fc5f4ff813a042d65e57ba7e0ccde44f40b78834
Oct 30 11:40:10.168582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.168551    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.168656 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.168619    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.168656 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.168650    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.191977 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.191952    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 11:40:10.202355 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:40:10.202309    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f4eaa7_4504_49b2_a03f_bc1e830a3828.slice/crio-293e7e2da1c7b2eb6e95d4bbdf59aba57a3455ef70d131cd1201d9e00828e511 WatchSource:0}: Error finding container 293e7e2da1c7b2eb6e95d4bbdf59aba57a3455ef70d131cd1201d9e00828e511: Status 404 returned error can't find the container with id 293e7e2da1c7b2eb6e95d4bbdf59aba57a3455ef70d131cd1201d9e00828e511
Oct 30 11:40:10.204526 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.204496    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.204627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.204567    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.204627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.204613    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:40:10.750906 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.750862    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:10.763495 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:10.763469    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:11.165552 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.165516    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" event={"ID":"65f4eaa7-4504-49b2-a03f-bc1e830a3828","Type":"ContainerStarted","Data":"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"}
Oct 30 11:40:11.165836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.165808    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" event={"ID":"65f4eaa7-4504-49b2-a03f-bc1e830a3828","Type":"ContainerStarted","Data":"293e7e2da1c7b2eb6e95d4bbdf59aba57a3455ef70d131cd1201d9e00828e511"}
Oct 30 11:40:11.167727 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.167701    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" event={"ID":"8718780e-49dd-4e5c-be31-527d4d6590df","Type":"ContainerStarted","Data":"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"}
Oct 30 11:40:11.167829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.167748    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" event={"ID":"8718780e-49dd-4e5c-be31-527d4d6590df","Type":"ContainerStarted","Data":"67d5a4f6eb0cac72a5932b94fc5f4ff813a042d65e57ba7e0ccde44f40b78834"}
Oct 30 11:40:11.180503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.180483    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-99czt"
Oct 30 11:40:11.267929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.267872    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podStartSLOduration=2.267857206 podStartE2EDuration="2.267857206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:11.212532103 +0000 UTC m=+610.432817013" watchObservedRunningTime="2026-10-30 11:40:11.267857206 +0000 UTC m=+610.488142116"
Oct 30 11:40:11.835903 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.835858    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:11.839693 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.839651    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:11.839834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.839730    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:11.872290 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.872262    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:11.873435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.873415    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:11.873525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:11.873454    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:12.754038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.753990    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:12.766465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.766437    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:12.810355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.809282    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podStartSLOduration=3.809265022 podStartE2EDuration="3.809265022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:11.324235243 +0000 UTC m=+610.544520167" watchObservedRunningTime="2026-10-30 11:40:12.809265022 +0000 UTC m=+612.029549934"
Oct 30 11:40:12.835354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.833692    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:12.835354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.833763    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:12.872606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.872576    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:12.872937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:12.872620    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:13.203353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:13.199554    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:40:13.839849 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:13.839807    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:13.840105 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:13.840075    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:13.872167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:13.872129    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:13.872387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:13.872215    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:14.836790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:14.836750    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:14.837399 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:14.837363    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:14.872759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:14.872732    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:14.872922 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:14.872903    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:15.834355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:15.833752    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:15.834355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:15.833820    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:15.872253 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:15.872218    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:15.872646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:15.872268    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:16.833167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:16.833132    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 11:40:16.833321 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:16.833187    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 11:40:16.871776 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:16.871746    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 11:40:16.871904 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:16.871789    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 11:40:17.012394 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.012323    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d59685b8f-lbdsm"]
Oct 30 11:40:17.016182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.016160    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.036194 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.036171    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d59685b8f-lbdsm"]
Oct 30 11:40:17.095307 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.095223    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593c626b-4093-4bc3-8b00-08c8a8c6af8d-tls-cert\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.095307 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.095267    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hwf\" (UniqueName: \"kubernetes.io/projected/593c626b-4093-4bc3-8b00-08c8a8c6af8d-kube-api-access-q2hwf\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.196659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.196630    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593c626b-4093-4bc3-8b00-08c8a8c6af8d-tls-cert\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.196659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.196663    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hwf\" (UniqueName: \"kubernetes.io/projected/593c626b-4093-4bc3-8b00-08c8a8c6af8d-kube-api-access-q2hwf\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.199179 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.199152    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593c626b-4093-4bc3-8b00-08c8a8c6af8d-tls-cert\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.216980 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.216956    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hwf\" (UniqueName: \"kubernetes.io/projected/593c626b-4093-4bc3-8b00-08c8a8c6af8d-kube-api-access-q2hwf\") pod \"authorino-5d59685b8f-lbdsm\" (UID: \"593c626b-4093-4bc3-8b00-08c8a8c6af8d\") " pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.328189 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.328141    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d59685b8f-lbdsm"
Oct 30 11:40:17.629675 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.629312    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d59685b8f-lbdsm"]
Oct 30 11:40:17.631760 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:40:17.631732    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod593c626b_4093_4bc3_8b00_08c8a8c6af8d.slice/crio-230e6efe51b791570aab1236a93da9859e25d32b666c5f955b3c536143994683 WatchSource:0}: Error finding container 230e6efe51b791570aab1236a93da9859e25d32b666c5f955b3c536143994683: Status 404 returned error can't find the container with id 230e6efe51b791570aab1236a93da9859e25d32b666c5f955b3c536143994683
Oct 30 11:40:17.633718 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:17.633696    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:40:18.204558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.204511    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d59685b8f-lbdsm" event={"ID":"593c626b-4093-4bc3-8b00-08c8a8c6af8d","Type":"ContainerStarted","Data":"230e6efe51b791570aab1236a93da9859e25d32b666c5f955b3c536143994683"}
Oct 30 11:40:18.502892 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.502781    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:18.503199 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.503174    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:18.504067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.504037    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 11:40:18.530364 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.530326    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:18.530645 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.530620    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:18.531457 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:18.531438    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:40:19.210417 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.210375    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d59685b8f-lbdsm" event={"ID":"593c626b-4093-4bc3-8b00-08c8a8c6af8d","Type":"ContainerStarted","Data":"ea1cc479f269d89828b2d0d63d30af6c811348eba7124d17239a19cbd8d86736"}
Oct 30 11:40:19.245204 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.245154    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5d59685b8f-lbdsm" podStartSLOduration=2.266627415 podStartE2EDuration="3.245137419s" totalImagesPullingTime="978.510004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:40:19.244233845 +0000 UTC m=+618.464518756" watchObservedRunningTime="2026-10-30 11:40:19.245137419 +0000 UTC m=+618.465422332"
Oct 30 11:40:19.269459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.269430    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:40:19.269622 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.269597    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" podUID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" containerName="authorino" containerID="cri-o://6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64" gracePeriod=30
Oct 30 11:40:19.513306 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.513279    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:40:19.616241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.616207    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\" (UniqueName: \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\") pod \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") "
Oct 30 11:40:19.616241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.616243    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\" (UniqueName: \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\") pod \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\" (UID: \"fe713ca6-b0aa-49c8-9496-ab12690e66ea\") "
Oct 30 11:40:19.618565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.618529    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6" pod "fe713ca6-b0aa-49c8-9496-ab12690e66ea" (UID: "fe713ca6-b0aa-49c8-9496-ab12690e66ea"). InnerVolumeSpecName "kube-api-access-rmck6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:40:19.626581 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.626556    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert" pod "fe713ca6-b0aa-49c8-9496-ab12690e66ea" (UID: "fe713ca6-b0aa-49c8-9496-ab12690e66ea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:40:19.717435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.717399    4453 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fe713ca6-b0aa-49c8-9496-ab12690e66ea-tls-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:40:19.717435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:19.717434    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmck6\" (UniqueName: \"kubernetes.io/projected/fe713ca6-b0aa-49c8-9496-ab12690e66ea-kube-api-access-rmck6\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:40:20.216506 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.216473    4453 generic.go:356] "Generic (PLEG): container finished" podID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" containerID="6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64" exitCode=0
Oct 30 11:40:20.216956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.216553    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" event={"ID":"fe713ca6-b0aa-49c8-9496-ab12690e66ea","Type":"ContainerDied","Data":"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"}
Oct 30 11:40:20.216956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.216573    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f9555bd6-jmjc6"
Oct 30 11:40:20.216956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.216602    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f9555bd6-jmjc6" event={"ID":"fe713ca6-b0aa-49c8-9496-ab12690e66ea","Type":"ContainerDied","Data":"bd1426adb7af3004a590f0a3cb09cc83a04cdd34ed50af916514624b7ee15006"}
Oct 30 11:40:20.216956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.216620    4453 scope.go:122] "RemoveContainer" containerID="6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"
Oct 30 11:40:20.225486 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.225469    4453 scope.go:122] "RemoveContainer" containerID="6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"
Oct 30 11:40:20.225746 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:40:20.225730    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64\": container with ID starting with 6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64 not found: ID does not exist" containerID="6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"
Oct 30 11:40:20.225789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.225754    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64"} err="failed to get container status \"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64\": rpc error: code = NotFound desc = could not find container \"6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64\": container with ID starting with 6e58a43f0c0c62173ed56517de66b82b2c227e9e62ca23f66c53367cc7eecf64 not found: ID does not exist"
Oct 30 11:40:20.264862 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.264825    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:40:20.274164 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.274137    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68f9555bd6-jmjc6"]
Oct 30 11:40:20.909854 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:20.909812    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" path="/var/lib/kubelet/pods/fe713ca6-b0aa-49c8-9496-ab12690e66ea/volumes"
Oct 30 11:40:24.978418 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:40:24.978374    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:40:37.570211 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:40:37.570168    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:41:07.569940 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:41:07.569893    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:41:37.570084 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:41:37.570005    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:41:38.862901 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.862869    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:41:38.863326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.863308    4453 cpu_manager.go:414] "containerMap: removing container" podUID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" containerName="authorino"
Oct 30 11:41:38.863326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.863340    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" containerName="authorino"
Oct 30 11:41:38.863499 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.863435    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe713ca6-b0aa-49c8-9496-ab12690e66ea" containerName="authorino"
Oct 30 11:41:38.866924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.866902    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.889616 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.889593    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0836c883-openshift-default-dockercfg-cv4cj\""
Oct 30 11:41:38.901612 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.901585    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:41:38.945450 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945422    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945465    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945491    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945616    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945679    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxg4\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945782    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945874 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945822    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.945874 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945859    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:38.946019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:38.945997    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.046802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.046764    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.046802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.046807    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.046832    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.046880    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxg4\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.046908    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047043    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047092    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047159    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047212    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047382    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047516 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047440    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047573 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047518    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047626 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047589    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.047777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.047756    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.049071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.049048    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.049285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.049268    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.078810 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.078785    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxg4\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.110070 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.110046    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\") pod \"e2e-ait-0836c883-openshift-default-56665d7c5-hthqp\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") " pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.181418 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.181340    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:39.279803 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.278127    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:41:39.283188 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.283159    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.324354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.324306    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:41:39.344608 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.344584    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3b30fbca-gw-openshift-default-dockercfg-cx659\""
Oct 30 11:41:39.350239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350204    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350425 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350254    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350425 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350356    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350425 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350411    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350610 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350483    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350610 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350510    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350610 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350597    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350766 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350649    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.350766 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.350679    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qrw\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451621    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451674    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451699    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qrw\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451725    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451743    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451797    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451859    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451888    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.451911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.451911    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.452353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.452135    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.452353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.452181    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.452353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.452230    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.452519 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.452405    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.452576 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.452528    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.454761 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.454739    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.454910 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.454832    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.473881 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.473848    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.474307 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.474284    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qrw\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\") pod \"e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") " pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.497209 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.497182    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:41:39.499319 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:39.499287    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20c9da5_9c97_4bdf_a7cf_13dfeee18eb8.slice/crio-ba21b692615621eaf3d6bbbd13b8f0c766accc84d82d4f566ea94780314d7c44 WatchSource:0}: Error finding container ba21b692615621eaf3d6bbbd13b8f0c766accc84d82d4f566ea94780314d7c44: Status 404 returned error can't find the container with id ba21b692615621eaf3d6bbbd13b8f0c766accc84d82d4f566ea94780314d7c44
Oct 30 11:41:39.501529 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.501491    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:39.501621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.501570    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:39.501668 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.501615    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:39.589701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.589664    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerStarted","Data":"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"}
Oct 30 11:41:39.589845 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.589727    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerStarted","Data":"ba21b692615621eaf3d6bbbd13b8f0c766accc84d82d4f566ea94780314d7c44"}
Oct 30 11:41:39.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.600025    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:39.633779 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.633243    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podStartSLOduration=1.63322279 podStartE2EDuration="1.63322279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:39.630383339 +0000 UTC m=+698.850668250" watchObservedRunningTime="2026-10-30 11:41:39.63322279 +0000 UTC m=+698.853507701"
Oct 30 11:41:39.912173 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.912143    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:41:39.913828 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:39.913801    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a812aa_b130_4137_93ef_6ec5f21f1ff3.slice/crio-615f5437825b33330849f4f16c9727f7a4f76c2600a51a661d8dd3c0ceda2016 WatchSource:0}: Error finding container 615f5437825b33330849f4f16c9727f7a4f76c2600a51a661d8dd3c0ceda2016: Status 404 returned error can't find the container with id 615f5437825b33330849f4f16c9727f7a4f76c2600a51a661d8dd3c0ceda2016
Oct 30 11:41:39.916174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.916142    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:39.916246 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.916203    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:39.916246 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:39.916233    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:40.181978 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.181889    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:40.186928 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.186904    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:40.597798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.597689    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerStarted","Data":"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"}
Oct 30 11:41:40.597798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.597752    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerStarted","Data":"615f5437825b33330849f4f16c9727f7a4f76c2600a51a661d8dd3c0ceda2016"}
Oct 30 11:41:40.598099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.598080    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:40.599399 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.599377    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:40.600467 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.600446    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:40.606437 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.606414    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:40.639682 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:40.639631    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podStartSLOduration=1.639617119 podStartE2EDuration="1.639617119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:40.638466903 +0000 UTC m=+699.858751825" watchObservedRunningTime="2026-10-30 11:41:40.639617119 +0000 UTC m=+699.859902028"
Oct 30 11:41:41.602700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:41.602588    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:41.603784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:41.603763    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:41:42.949627 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.949594    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:41:42.953842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.953808    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:42.969652 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.969525    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0836c883-serving-cert\""
Oct 30 11:41:42.977575 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.977534    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:41:42.991604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.991382    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:42.991604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.991504    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqn8t\" (UniqueName: \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:42.991604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.991544    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:42.991604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:42.991586    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.093090 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.093058    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.093253 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.093133    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.093253 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.093183    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqn8t\" (UniqueName: \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.093253 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.093221    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.093995 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.093975    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.095520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.095497    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.095676 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.095657    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.126664 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.126633    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqn8t\" (UniqueName: \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\") pod \"maas-api-e2e-ait-0836c883-8695bd865f-m52dr\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.269692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.269607    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:43.597117 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.597022    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:41:43.600993 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:43.600968    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e58a8ec_e82e_4c10_931d_4ccf776d7b82.slice/crio-9789e99e87ca0e256d83ece2bf30ab2dda03cf521378ab4f4b96091e39f46e76 WatchSource:0}: Error finding container 9789e99e87ca0e256d83ece2bf30ab2dda03cf521378ab4f4b96091e39f46e76: Status 404 returned error can't find the container with id 9789e99e87ca0e256d83ece2bf30ab2dda03cf521378ab4f4b96091e39f46e76
Oct 30 11:41:43.616310 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:43.616284    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" event={"ID":"4e58a8ec-e82e-4c10-931d-4ccf776d7b82","Type":"ContainerStarted","Data":"9789e99e87ca0e256d83ece2bf30ab2dda03cf521378ab4f4b96091e39f46e76"}
Oct 30 11:41:44.622354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:44.622293    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" event={"ID":"4e58a8ec-e82e-4c10-931d-4ccf776d7b82","Type":"ContainerStarted","Data":"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"}
Oct 30 11:41:44.622742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:44.622393    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:44.662605 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:44.662547    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" podStartSLOduration=2.245125614 podStartE2EDuration="2.662533203s" totalImagesPullingTime="417.407589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:44.661038528 +0000 UTC m=+703.881323438" watchObservedRunningTime="2026-10-30 11:41:44.662533203 +0000 UTC m=+703.882818113"
Oct 30 11:41:45.230809 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.230772    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:41:45.235028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.235005    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.254799 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.254770    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:41:45.259319 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.259295    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3b30fbca-dockercfg-ks495\""
Oct 30 11:41:45.259319 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.259299    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3b30fbca\""
Oct 30 11:41:45.319467 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.319437    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlll4\" (UniqueName: \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.319583 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.319557    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.379819 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.379786    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:41:45.383435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.383413    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.400309 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.400292    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3b30fbca-serving-cert\""
Oct 30 11:41:45.406914 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.406870    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:41:45.420883 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.420853    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpj5\" (UniqueName: \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.421033 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.420961    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlll4\" (UniqueName: \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.421108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.421032    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.421108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.421101    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.421226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.421145    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.421226 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.421186    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.421955 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.421934    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.443141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.443111    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlll4\" (UniqueName: \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\") pod \"payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") " pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.522954 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.522856    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpj5\" (UniqueName: \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.523109 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.523030    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.523109 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.523100    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.523212 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.523150    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.523968 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.523943    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.525602 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.525572    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.525749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.525733    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.543670 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.543648    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpj5\" (UniqueName: \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\") pod \"maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.548993 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.548972    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:45.695465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.695430    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:45.849419 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.849384    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:41:45.851343 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:45.851302    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc98ddb0_ce53_464e_a042_68a92d2e4ede.slice/crio-de5ef0ccfb697febfc4f210e8edc07d9d306275f5f5594b1b333b81f109b9e8c WatchSource:0}: Error finding container de5ef0ccfb697febfc4f210e8edc07d9d306275f5f5594b1b333b81f109b9e8c: Status 404 returned error can't find the container with id de5ef0ccfb697febfc4f210e8edc07d9d306275f5f5594b1b333b81f109b9e8c
Oct 30 11:41:45.990661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:45.990625    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:41:45.992418 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:45.992390    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aca7cf_391d_429e_a624_b1fef7feca0a.slice/crio-81ad4b5bbfab2b12871a7c733588ecccfcda7895d1b4e28f44922ef415c8aa7b WatchSource:0}: Error finding container 81ad4b5bbfab2b12871a7c733588ecccfcda7895d1b4e28f44922ef415c8aa7b: Status 404 returned error can't find the container with id 81ad4b5bbfab2b12871a7c733588ecccfcda7895d1b4e28f44922ef415c8aa7b
Oct 30 11:41:46.641017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.640983    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" event={"ID":"dc98ddb0-ce53-464e-a042-68a92d2e4ede","Type":"ContainerStarted","Data":"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"}
Oct 30 11:41:46.641193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.641036    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" event={"ID":"dc98ddb0-ce53-464e-a042-68a92d2e4ede","Type":"ContainerStarted","Data":"de5ef0ccfb697febfc4f210e8edc07d9d306275f5f5594b1b333b81f109b9e8c"}
Oct 30 11:41:46.641193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.641098    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:46.642591 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.642563    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" event={"ID":"91aca7cf-391d-429e-a624-b1fef7feca0a","Type":"ContainerStarted","Data":"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"}
Oct 30 11:41:46.642705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.642607    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:46.642705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.642626    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" event={"ID":"91aca7cf-391d-429e-a624-b1fef7feca0a","Type":"ContainerStarted","Data":"81ad4b5bbfab2b12871a7c733588ecccfcda7895d1b4e28f44922ef415c8aa7b"}
Oct 30 11:41:46.675549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.675504    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" podStartSLOduration=1.67549209 podStartE2EDuration="1.67549209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:46.675095753 +0000 UTC m=+705.895380664" watchObservedRunningTime="2026-10-30 11:41:46.67549209 +0000 UTC m=+705.895777000"
Oct 30 11:41:46.712549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:46.712502    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" podStartSLOduration=1.401270901 podStartE2EDuration="1.712491457s" totalImagesPullingTime="311.220556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:46.712389596 +0000 UTC m=+705.932674506" watchObservedRunningTime="2026-10-30 11:41:46.712491457 +0000 UTC m=+705.932776366"
Oct 30 11:41:49.303360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.301519    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:41:49.313350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.309574    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.323286 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.323263    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:41:49.337362 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.336976    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-14a5c09a-serving-cert\""
Oct 30 11:41:49.363510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.363481    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.363781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.363762    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2nt\" (UniqueName: \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.363926 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.363913    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.364121 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.364108    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.465141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.465100    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.465315 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.465193    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.465315 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.465249    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.465315 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.465275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2nt\" (UniqueName: \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.465975 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.465950    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.467748 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.467727    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.467847 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.467824    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.485509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.485485    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2nt\" (UniqueName: \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\") pod \"maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.625803 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.625768    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:49.926443 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:49.926169    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:41:49.928517 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:49.928490    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef64a37c_a538_4fc6_be88_014d2f3ae2b5.slice/crio-88be21572ab6fe0154d78cc85b29824a0b702b8472845374132c303b01ead293 WatchSource:0}: Error finding container 88be21572ab6fe0154d78cc85b29824a0b702b8472845374132c303b01ead293: Status 404 returned error can't find the container with id 88be21572ab6fe0154d78cc85b29824a0b702b8472845374132c303b01ead293
Oct 30 11:41:50.643374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:50.641113    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:41:50.669814 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:50.669757    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" event={"ID":"ef64a37c-a538-4fc6-be88-014d2f3ae2b5","Type":"ContainerStarted","Data":"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"}
Oct 30 11:41:50.670006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:50.669847    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:50.670006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:50.669871    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" event={"ID":"ef64a37c-a538-4fc6-be88-014d2f3ae2b5","Type":"ContainerStarted","Data":"88be21572ab6fe0154d78cc85b29824a0b702b8472845374132c303b01ead293"}
Oct 30 11:41:50.758717 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:50.758021    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" podStartSLOduration=1.41744641 podStartE2EDuration="1.758003335s" totalImagesPullingTime="340.556925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:50.756254276 +0000 UTC m=+709.976539187" watchObservedRunningTime="2026-10-30 11:41:50.758003335 +0000 UTC m=+709.978288247"
Oct 30 11:41:52.658918 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:52.658890    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:41:55.835416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.835382    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:41:55.839494 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.839474    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:55.863659 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.863617    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-90364398-dockercfg-59gb2\""
Oct 30 11:41:55.864555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.864532    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-90364398\""
Oct 30 11:41:55.867487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.867462    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:41:55.925384 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.924320    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:55.925841 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.925808    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzk9\" (UniqueName: \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:55.938391 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.931012    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:41:55.962459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.959550    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:41:55.962459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:55.959684    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.027286 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.027246    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.027505 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.027309    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzk9\" (UniqueName: \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.027580 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.027508    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.027580 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.027566    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8kr\" (UniqueName: \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.028181 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.028161    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.049088 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.049058    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzk9\" (UniqueName: \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\") pod \"payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.128790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.128754    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8kr\" (UniqueName: \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.128990 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.128844    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.129616 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.129589    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.150012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.149980    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8kr\" (UniqueName: \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\") pod \"payload-processing-e2e-ait-90364398-77778cb55d-d7kzd\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") " pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.157025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.157002    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.277103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.277069    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.481603 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.481568    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:41:56.485528 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:56.485488    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422215a3_12eb_4976_8e62_d6c8a2856c31.slice/crio-1831d5bec53bcf4997286d1d5bdbe1d1abe54cb744a92ee59867d57be779fda4 WatchSource:0}: Error finding container 1831d5bec53bcf4997286d1d5bdbe1d1abe54cb744a92ee59867d57be779fda4: Status 404 returned error can't find the container with id 1831d5bec53bcf4997286d1d5bdbe1d1abe54cb744a92ee59867d57be779fda4
Oct 30 11:41:56.615069 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.615035    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:41:56.618881 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:41:56.618848    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23545519_1f4c_44c8_bd0f_7a4dae434f5a.slice/crio-ff9db0e1a76334c94f6260839d24dccc03e6951925a374038ee72b8494c97b11 WatchSource:0}: Error finding container ff9db0e1a76334c94f6260839d24dccc03e6951925a374038ee72b8494c97b11: Status 404 returned error can't find the container with id ff9db0e1a76334c94f6260839d24dccc03e6951925a374038ee72b8494c97b11
Oct 30 11:41:56.643499 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.643464    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:41:56.643592 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.643517    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:41:56.682089 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.682065    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:41:56.728099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.728051    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" event={"ID":"23545519-1f4c-44c8-bd0f-7a4dae434f5a","Type":"ContainerStarted","Data":"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"}
Oct 30 11:41:56.728289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.728121    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" event={"ID":"23545519-1f4c-44c8-bd0f-7a4dae434f5a","Type":"ContainerStarted","Data":"ff9db0e1a76334c94f6260839d24dccc03e6951925a374038ee72b8494c97b11"}
Oct 30 11:41:56.728289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.728233    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:41:56.730844 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.730809    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" event={"ID":"422215a3-12eb-4976-8e62-d6c8a2856c31","Type":"ContainerStarted","Data":"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"}
Oct 30 11:41:56.730983 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.730888    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" event={"ID":"422215a3-12eb-4976-8e62-d6c8a2856c31","Type":"ContainerStarted","Data":"1831d5bec53bcf4997286d1d5bdbe1d1abe54cb744a92ee59867d57be779fda4"}
Oct 30 11:41:56.731042 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.731024    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:41:56.769443 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.769315    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" podStartSLOduration=1.7692963910000001 podStartE2EDuration="1.769296391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:56.766262188 +0000 UTC m=+715.986547099" watchObservedRunningTime="2026-10-30 11:41:56.769296391 +0000 UTC m=+715.989581304"
Oct 30 11:41:56.806662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:56.806599    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" podStartSLOduration=1.8065789639999998 podStartE2EDuration="1.806578964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:41:56.804493179 +0000 UTC m=+716.024778088" watchObservedRunningTime="2026-10-30 11:41:56.806578964 +0000 UTC m=+716.026863876"
Oct 30 11:41:57.643532 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.643490    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:41:57.643992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.643555    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:41:57.656788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.656762    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:41:57.738434 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.738397    4453 generic.go:356] "Generic (PLEG): container finished" podID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerID="7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd" exitCode=0
Oct 30 11:41:57.738601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.738473    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerDied","Data":"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"}
Oct 30 11:41:57.739102 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.738994    4453 scope.go:122] "RemoveContainer" containerID="7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"
Oct 30 11:41:57.740411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.740379    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:57.740518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.740468    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:57.740559 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:57.740521    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:41:58.745069 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:58.745037    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerStarted","Data":"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"}
Oct 30 11:41:59.182164 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.182131    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:41:59.182411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.182384    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 11:41:59.182485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.182447    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 11:41:59.842207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.842164    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:41:59.847214 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.847195    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:41:59.859794 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.859750    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-64715175-serving-cert\""
Oct 30 11:41:59.872021 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.871995    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:41:59.977259 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.977148    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:41:59.977259 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.977225    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:41:59.977464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.977269    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:41:59.977661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:41:59.977638    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4mc\" (UniqueName: \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.078639 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.078601    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.078831 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.078686    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.078831 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.078732    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.078831 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.078778    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4mc\" (UniqueName: \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.079275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.079238    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.081222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.081195    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.081351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.081250    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.112360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.111110    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4mc\" (UniqueName: \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\") pod \"maas-api-e2e-derive-64715175-958c45dbc-5qxhq\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.160542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.160504    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:00.181988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.181958    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 11:42:00.182126 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.182028    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 11:42:00.480301 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.480272    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:42:00.482139 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:00.482082    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5896fc64_ade9_4326_9eed_caf0ca3b52f0.slice/crio-d9ac046b92492543e4c8e5b7d36b060d9c8108550ad9f70e51212099c7a36547 WatchSource:0}: Error finding container d9ac046b92492543e4c8e5b7d36b060d9c8108550ad9f70e51212099c7a36547: Status 404 returned error can't find the container with id d9ac046b92492543e4c8e5b7d36b060d9c8108550ad9f70e51212099c7a36547
Oct 30 11:42:00.761485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.761405    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" event={"ID":"5896fc64-ade9-4326-9eed-caf0ca3b52f0","Type":"ContainerStarted","Data":"d9ac046b92492543e4c8e5b7d36b060d9c8108550ad9f70e51212099c7a36547"}
Oct 30 11:42:00.763210 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.763183    4453 generic.go:356] "Generic (PLEG): container finished" podID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerID="3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47" exitCode=0
Oct 30 11:42:00.763382 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.763257    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerDied","Data":"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"}
Oct 30 11:42:00.763702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.763686    4453 scope.go:122] "RemoveContainer" containerID="3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"
Oct 30 11:42:00.771287 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.771234    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:00.771402 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.771305    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:00.771402 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:00.771354    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:01.182591 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.182561    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 11:42:01.183072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.182609    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 11:42:01.776118 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.776078    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" event={"ID":"5896fc64-ade9-4326-9eed-caf0ca3b52f0","Type":"ContainerStarted","Data":"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"}
Oct 30 11:42:01.776314 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.776199    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:01.778148 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.778124    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerStarted","Data":"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"}
Oct 30 11:42:01.812355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.809587    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:01.860005 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.858477    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:01.860005 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.858610    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:01.871241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.871221    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-90364398-serving-cert\""
Oct 30 11:42:01.890601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.890545    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" podStartSLOduration=2.599454933 podStartE2EDuration="2.89052494s" totalImagesPullingTime="291.070007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:01.846973997 +0000 UTC m=+721.067258909" watchObservedRunningTime="2026-10-30 11:42:01.89052494 +0000 UTC m=+721.110809865"
Oct 30 11:42:01.949370 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.947910    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:01.949577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.949529    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:01.949728 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.949704    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:01.949794 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:01.949743    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n584\" (UniqueName: \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.051174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.051073    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.051387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.051257    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.051387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.051305    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n584\" (UniqueName: \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.051493 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.051406    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.053179 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.053156    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.054551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.054531    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.054694 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.054647    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.072782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.072754    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n584\" (UniqueName: \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\") pod \"maas-api-e2e-ait-90364398-965944996-6kptw\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.176897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.176860    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:02.182446 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.182412    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 11:42:02.182578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.182477    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 11:42:02.584723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.584637    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:02.615367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.601722    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:02.615367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.606211    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 11:42:02.615367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.606261    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 11:42:02.784994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:02.784960    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" event={"ID":"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a","Type":"ContainerStarted","Data":"e40addc51b16337909d6cb205022242a7b05165662f920da38bac78a2b3af060"}
Oct 30 11:42:03.182030 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.181990    4453 patch_prober.go:29] interesting pod/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 11:42:03.182295 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.182267    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 11:42:03.624355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.609890    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 11:42:03.624355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.609966    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 11:42:03.810359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.810176    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" event={"ID":"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a","Type":"ContainerStarted","Data":"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"}
Oct 30 11:42:03.811358 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.811171    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:03.850428 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:03.850363    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" podStartSLOduration=2.500286439 podStartE2EDuration="2.850319314s" totalImagesPullingTime="350.032875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:03.849134381 +0000 UTC m=+723.069419292" watchObservedRunningTime="2026-10-30 11:42:03.850319314 +0000 UTC m=+723.070604223"
Oct 30 11:42:04.187510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:04.187479    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:42:04.187920 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:04.187891    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:42:04.205687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:04.205656    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:42:04.611359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:04.607966    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 11:42:04.611359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:04.608141    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 11:42:05.606829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:05.606785    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 11:42:05.607876 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:05.607445    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 11:42:06.607793 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:06.607751    4453 patch_prober.go:29] interesting pod/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 11:42:06.608424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:06.608390    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 11:42:07.572574 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:07.572537    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:42:07.604585 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.604556    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:07.604914 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.604888    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:07.605616 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.605600    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:07.741673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.741643    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:42:07.742123 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.742013    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:42:07.790972 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.790941    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:07.993080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:07.993044    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:42:10.831080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:10.831051    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:12.994233 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:12.994183    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" podUID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" containerName="payload-processing" containerID="cri-o://5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630" gracePeriod=25
Oct 30 11:42:13.242625 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.242600    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:42:13.297926 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.297841    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\" (UniqueName: \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\") pod \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") "
Oct 30 11:42:13.298092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.297939    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\") pod \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\" (UID: \"dc98ddb0-ce53-464e-a042-68a92d2e4ede\") "
Oct 30 11:42:13.298275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.298253    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume" pod "dc98ddb0-ce53-464e-a042-68a92d2e4ede" (UID: "dc98ddb0-ce53-464e-a042-68a92d2e4ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:13.298411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.298394    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc98ddb0-ce53-464e-a042-68a92d2e4ede-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:13.299963 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.299932    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4" pod "dc98ddb0-ce53-464e-a042-68a92d2e4ede" (UID: "dc98ddb0-ce53-464e-a042-68a92d2e4ede"). InnerVolumeSpecName "kube-api-access-vlll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:13.399090 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.399047    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlll4\" (UniqueName: \"kubernetes.io/projected/dc98ddb0-ce53-464e-a042-68a92d2e4ede-kube-api-access-vlll4\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:13.869852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.869809    4453 generic.go:356] "Generic (PLEG): container finished" podID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" containerID="5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630" exitCode=0
Oct 30 11:42:13.870048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.869865    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"
Oct 30 11:42:13.870048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.869887    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" event={"ID":"dc98ddb0-ce53-464e-a042-68a92d2e4ede","Type":"ContainerDied","Data":"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"}
Oct 30 11:42:13.870048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.869952    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv" event={"ID":"dc98ddb0-ce53-464e-a042-68a92d2e4ede","Type":"ContainerDied","Data":"de5ef0ccfb697febfc4f210e8edc07d9d306275f5f5594b1b333b81f109b9e8c"}
Oct 30 11:42:13.870048 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.869983    4453 scope.go:122] "RemoveContainer" containerID="5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"
Oct 30 11:42:13.879083 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.879066    4453 scope.go:122] "RemoveContainer" containerID="5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"
Oct 30 11:42:13.879409 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:13.879387    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630\": container with ID starting with 5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630 not found: ID does not exist" containerID="5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"
Oct 30 11:42:13.879476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.879419    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630"} err="failed to get container status \"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630\": rpc error: code = NotFound desc = could not find container \"5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630\": container with ID starting with 5e4457c3ad8220ba87369e23ebdcb646c0fe6b50b8acad53b571e55b76a8a630 not found: ID does not exist"
Oct 30 11:42:13.913503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.913471    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:42:13.922770 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:13.922748    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3b30fbca-c566f6dfb-gxztv"]
Oct 30 11:42:14.910581 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:14.910533    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" path="/var/lib/kubelet/pods/dc98ddb0-ce53-464e-a042-68a92d2e4ede/volumes"
Oct 30 11:42:15.615218 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.615186    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:42:15.615450 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.615407    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerName="maas-api" containerID="cri-o://e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287" gracePeriod=30
Oct 30 11:42:15.635385 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.635348    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.41:8443/health\": dial tcp 10.133.0.41:8443: connect: connection refused"
Oct 30 11:42:15.862107 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.862072    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:42:15.882981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.882888    4453 generic.go:356] "Generic (PLEG): container finished" podID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerID="e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287" exitCode=0
Oct 30 11:42:15.882981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.882930    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"
Oct 30 11:42:15.882981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.882960    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" event={"ID":"4e58a8ec-e82e-4c10-931d-4ccf776d7b82","Type":"ContainerDied","Data":"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"}
Oct 30 11:42:15.883218 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.883019    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr" event={"ID":"4e58a8ec-e82e-4c10-931d-4ccf776d7b82","Type":"ContainerDied","Data":"9789e99e87ca0e256d83ece2bf30ab2dda03cf521378ab4f4b96091e39f46e76"}
Oct 30 11:42:15.883218 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.883051    4453 scope.go:122] "RemoveContainer" containerID="e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"
Oct 30 11:42:15.892178 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.892162    4453 scope.go:122] "RemoveContainer" containerID="e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"
Oct 30 11:42:15.892476 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:15.892457    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287\": container with ID starting with e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287 not found: ID does not exist" containerID="e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"
Oct 30 11:42:15.892525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.892489    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287"} err="failed to get container status \"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287\": rpc error: code = NotFound desc = could not find container \"e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287\": container with ID starting with e664c3dac0fc1f6ac5479f0783ff93d2f8e9ff4bb6dabfd53e538491f9e2c287 not found: ID does not exist"
Oct 30 11:42:15.924667 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.924637    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\") pod \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") "
Oct 30 11:42:15.925129 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.924685    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\") pod \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") "
Oct 30 11:42:15.925129 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.924759    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\" (UniqueName: \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\") pod \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") "
Oct 30 11:42:15.925129 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.924845    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\") pod \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\" (UID: \"4e58a8ec-e82e-4c10-931d-4ccf776d7b82\") "
Oct 30 11:42:15.925357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.925264    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca" pod "4e58a8ec-e82e-4c10-931d-4ccf776d7b82" (UID: "4e58a8ec-e82e-4c10-931d-4ccf776d7b82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:15.926977 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.926950    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls" pod "4e58a8ec-e82e-4c10-931d-4ccf776d7b82" (UID: "4e58a8ec-e82e-4c10-931d-4ccf776d7b82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:15.927092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.927070    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t" pod "4e58a8ec-e82e-4c10-931d-4ccf776d7b82" (UID: "4e58a8ec-e82e-4c10-931d-4ccf776d7b82"). InnerVolumeSpecName "kube-api-access-zqn8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:15.927176 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:15.927158    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs" pod "4e58a8ec-e82e-4c10-931d-4ccf776d7b82" (UID: "4e58a8ec-e82e-4c10-931d-4ccf776d7b82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:16.025846 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.025804    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.025846 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.025843    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.025846 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.025857    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.026108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.025868    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqn8t\" (UniqueName: \"kubernetes.io/projected/4e58a8ec-e82e-4c10-931d-4ccf776d7b82-kube-api-access-zqn8t\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.189357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.189238    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:42:16.189514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.189446    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" podUID="91aca7cf-391d-429e-a624-b1fef7feca0a" containerName="maas-api" containerID="cri-o://280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7" gracePeriod=30
Oct 30 11:42:16.267181 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.267147    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:42:16.269504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.269467    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0836c883-8695bd865f-m52dr"]
Oct 30 11:42:16.447990 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.447962    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:42:16.536296 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.536261    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\" (UniqueName: \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\") pod \"91aca7cf-391d-429e-a624-b1fef7feca0a\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") "
Oct 30 11:42:16.536296 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.536300    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\") pod \"91aca7cf-391d-429e-a624-b1fef7feca0a\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") "
Oct 30 11:42:16.536565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.536350    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\") pod \"91aca7cf-391d-429e-a624-b1fef7feca0a\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") "
Oct 30 11:42:16.536565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.536433    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\") pod \"91aca7cf-391d-429e-a624-b1fef7feca0a\" (UID: \"91aca7cf-391d-429e-a624-b1fef7feca0a\") "
Oct 30 11:42:16.536823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.536798    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca" pod "91aca7cf-391d-429e-a624-b1fef7feca0a" (UID: "91aca7cf-391d-429e-a624-b1fef7feca0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:16.538672 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.538648    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls" pod "91aca7cf-391d-429e-a624-b1fef7feca0a" (UID: "91aca7cf-391d-429e-a624-b1fef7feca0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:16.538768 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.538665    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs" pod "91aca7cf-391d-429e-a624-b1fef7feca0a" (UID: "91aca7cf-391d-429e-a624-b1fef7feca0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:16.538768 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.538738    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5" pod "91aca7cf-391d-429e-a624-b1fef7feca0a" (UID: "91aca7cf-391d-429e-a624-b1fef7feca0a"). InnerVolumeSpecName "kube-api-access-snpj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:16.637479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.637448    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpj5\" (UniqueName: \"kubernetes.io/projected/91aca7cf-391d-429e-a624-b1fef7feca0a-kube-api-access-snpj5\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.637479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.637478    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.637653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.637488    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91aca7cf-391d-429e-a624-b1fef7feca0a-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.637653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.637496    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91aca7cf-391d-429e-a624-b1fef7feca0a-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:16.888969 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.888936    4453 generic.go:356] "Generic (PLEG): container finished" podID="91aca7cf-391d-429e-a624-b1fef7feca0a" containerID="280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7" exitCode=0
Oct 30 11:42:16.889153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.888982    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"
Oct 30 11:42:16.889153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.889027    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" event={"ID":"91aca7cf-391d-429e-a624-b1fef7feca0a","Type":"ContainerDied","Data":"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"}
Oct 30 11:42:16.889153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.889091    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r" event={"ID":"91aca7cf-391d-429e-a624-b1fef7feca0a","Type":"ContainerDied","Data":"81ad4b5bbfab2b12871a7c733588ecccfcda7895d1b4e28f44922ef415c8aa7b"}
Oct 30 11:42:16.889153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.889125    4453 scope.go:122] "RemoveContainer" containerID="280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"
Oct 30 11:42:16.898598 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.898583    4453 scope.go:122] "RemoveContainer" containerID="280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"
Oct 30 11:42:16.898828 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:16.898809    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7\": container with ID starting with 280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7 not found: ID does not exist" containerID="280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"
Oct 30 11:42:16.898875 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.898837    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7"} err="failed to get container status \"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7\": rpc error: code = NotFound desc = could not find container \"280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7\": container with ID starting with 280114e3b0f6c2eb35fb3c18a4b45997e4df2e8d830acb3fdafc2f588764b2c7 not found: ID does not exist"
Oct 30 11:42:16.910488 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.910463    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" path="/var/lib/kubelet/pods/4e58a8ec-e82e-4c10-931d-4ccf776d7b82/volumes"
Oct 30 11:42:16.931908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.931875    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:42:16.943649 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:16.943625    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3b30fbca-7c56fbfcf5-79w9r"]
Oct 30 11:42:17.209067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:17.208973    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:42:17.209216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:17.209177    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy" containerID="cri-o://a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35" gracePeriod=30
Oct 30 11:42:18.910218 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:18.910179    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aca7cf-391d-429e-a624-b1fef7feca0a" path="/var/lib/kubelet/pods/91aca7cf-391d-429e-a624-b1fef7feca0a/volumes"
Oct 30 11:42:18.910619 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:18.910504    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:42:18.910704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:18.910687    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" podUID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" containerName="maas-api" containerID="cri-o://c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795" gracePeriod=30
Oct 30 11:42:19.172996 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.172932    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:42:19.260701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.260674    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\" (UniqueName: \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\") pod \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") "
Oct 30 11:42:19.260859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.260734    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\") pod \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") "
Oct 30 11:42:19.260859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.260779    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\") pod \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") "
Oct 30 11:42:19.260859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.260825    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\") pod \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\" (UID: \"ef64a37c-a538-4fc6-be88-014d2f3ae2b5\") "
Oct 30 11:42:19.261126 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.261093    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca" pod "ef64a37c-a538-4fc6-be88-014d2f3ae2b5" (UID: "ef64a37c-a538-4fc6-be88-014d2f3ae2b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:19.262886 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.262857    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs" pod "ef64a37c-a538-4fc6-be88-014d2f3ae2b5" (UID: "ef64a37c-a538-4fc6-be88-014d2f3ae2b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:19.262886 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.262863    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt" pod "ef64a37c-a538-4fc6-be88-014d2f3ae2b5" (UID: "ef64a37c-a538-4fc6-be88-014d2f3ae2b5"). InnerVolumeSpecName "kube-api-access-nb2nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:19.262997 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.262888    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls" pod "ef64a37c-a538-4fc6-be88-014d2f3ae2b5" (UID: "ef64a37c-a538-4fc6-be88-014d2f3ae2b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:19.361806 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.361770    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2nt\" (UniqueName: \"kubernetes.io/projected/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-kube-api-access-nb2nt\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:19.361806 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.361795    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:19.361806 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.361805    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:19.361806 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.361813    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef64a37c-a538-4fc6-be88-014d2f3ae2b5-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:19.907369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.907306    4453 generic.go:356] "Generic (PLEG): container finished" podID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" containerID="c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795" exitCode=0
Oct 30 11:42:19.907369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.907370    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"
Oct 30 11:42:19.907621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.907381    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" event={"ID":"ef64a37c-a538-4fc6-be88-014d2f3ae2b5","Type":"ContainerDied","Data":"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"}
Oct 30 11:42:19.907621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.907435    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c" event={"ID":"ef64a37c-a538-4fc6-be88-014d2f3ae2b5","Type":"ContainerDied","Data":"88be21572ab6fe0154d78cc85b29824a0b702b8472845374132c303b01ead293"}
Oct 30 11:42:19.907621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.907458    4453 scope.go:122] "RemoveContainer" containerID="c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"
Oct 30 11:42:19.916863 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.916687    4453 scope.go:122] "RemoveContainer" containerID="c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"
Oct 30 11:42:19.917085 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:19.916947    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795\": container with ID starting with c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795 not found: ID does not exist" containerID="c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"
Oct 30 11:42:19.917085 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.916971    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795"} err="failed to get container status \"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795\": rpc error: code = NotFound desc = could not find container \"c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795\": container with ID starting with c4a8a498eab0c80cc09e7aff69dcac515fd9f66944609fe955b2ab4ddd30e795 not found: ID does not exist"
Oct 30 11:42:19.950188 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.950164    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:42:19.959676 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:19.959656    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14a5c09a-5cc454f986-sf95c"]
Oct 30 11:42:20.910775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:20.910742    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" path="/var/lib/kubelet/pods/ef64a37c-a538-4fc6-be88-014d2f3ae2b5/volumes"
Oct 30 11:42:22.458425 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.458401    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:22.592921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.592842    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.592921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.592885    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.592921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.592911    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.592960    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.592995    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593028    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593103    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593262    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593523 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593464    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\") pod \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\" (UID: \"23a812aa-b130-4137-93ef-6ec5f21f1ff3\") "
Oct 30 11:42:22.593758 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593692    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:22.593899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593772    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:22.593899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593862    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:22.594039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.593970    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.594039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.594008    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.594039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.594022    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.594192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.594136    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:22.594345 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.594232    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:22.595582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.595557    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "kube-api-access-s5qrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:22.595855 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.595838    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:22.595919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.595852    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:42:22.595981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.595926    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token" pod "23a812aa-b130-4137-93ef-6ec5f21f1ff3" (UID: "23a812aa-b130-4137-93ef-6ec5f21f1ff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:22.694852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694822    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.694852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694848    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.694852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694858    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.695053 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694867    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a812aa-b130-4137-93ef-6ec5f21f1ff3-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.695053 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694877    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qrw\" (UniqueName: \"kubernetes.io/projected/23a812aa-b130-4137-93ef-6ec5f21f1ff3-kube-api-access-s5qrw\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.695053 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.694887    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a812aa-b130-4137-93ef-6ec5f21f1ff3-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:22.923819 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.923793    4453 generic.go:356] "Generic (PLEG): container finished" podID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerID="a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35" exitCode=0
Oct 30 11:42:22.923938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.923830    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"
Oct 30 11:42:22.923938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.923869    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerDied","Data":"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"}
Oct 30 11:42:22.923938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.923916    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz" event={"ID":"23a812aa-b130-4137-93ef-6ec5f21f1ff3","Type":"ContainerDied","Data":"615f5437825b33330849f4f16c9727f7a4f76c2600a51a661d8dd3c0ceda2016"}
Oct 30 11:42:22.923938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.923937    4453 scope.go:122] "RemoveContainer" containerID="a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"
Oct 30 11:42:22.932423 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.932406    4453 scope.go:122] "RemoveContainer" containerID="3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"
Oct 30 11:42:22.940066 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.940051    4453 scope.go:122] "RemoveContainer" containerID="a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"
Oct 30 11:42:22.940294 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:22.940276    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35\": container with ID starting with a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35 not found: ID does not exist" containerID="a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"
Oct 30 11:42:22.940364 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.940300    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35"} err="failed to get container status \"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35\": rpc error: code = NotFound desc = could not find container \"a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35\": container with ID starting with a55480246c9404ebbb65925522d08fe8626e2fe8562de46a979ee514b94cfa35 not found: ID does not exist"
Oct 30 11:42:22.940364 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.940316    4453 scope.go:122] "RemoveContainer" containerID="3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"
Oct 30 11:42:22.940591 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:22.940574    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47\": container with ID starting with 3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47 not found: ID does not exist" containerID="3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"
Oct 30 11:42:22.940633 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.940597    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47"} err="failed to get container status \"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47\": rpc error: code = NotFound desc = could not find container \"3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47\": container with ID starting with 3b40f2ca2ec536eb2e176fdd12dea504d6ce14e40ab0b634f425c3934817fc47 not found: ID does not exist"
Oct 30 11:42:22.967733 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.967711    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:42:22.976994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:22.976973    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3b30fbca-gw-openshift-default-746d97988f-q8tpz"]
Oct 30 11:42:23.955778 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.955746    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 11:42:23.956305 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956287    4453 cpu_manager.go:414] "containerMap: removing container" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956307    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956320    4453 cpu_manager.go:414] "containerMap: removing container" podUID="91aca7cf-391d-429e-a624-b1fef7feca0a" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956326    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aca7cf-391d-429e-a624-b1fef7feca0a" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956349    4453 cpu_manager.go:414] "containerMap: removing container" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956354    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956361    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956366    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" containerName="maas-api"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956377    4453 cpu_manager.go:414] "containerMap: removing container" podUID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" containerName="payload-processing"
Oct 30 11:42:23.956379 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956381    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" containerName="payload-processing"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956401    4453 cpu_manager.go:414] "containerMap: removing container" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956406    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956466    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956475    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aca7cf-391d-429e-a624-b1fef7feca0a" containerName="maas-api"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956482    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e58a8ec-e82e-4c10-931d-4ccf776d7b82" containerName="maas-api"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956488    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef64a37c-a538-4fc6-be88-014d2f3ae2b5" containerName="maas-api"
Oct 30 11:42:23.956663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.956494    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc98ddb0-ce53-464e-a042-68a92d2e4ede" containerName="payload-processing"
Oct 30 11:42:23.961071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.961050    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:23.976525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.976507    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4e261b\""
Oct 30 11:42:23.984621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.984598    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4e261b-dockercfg-dfj5v\""
Oct 30 11:42:23.989668 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:23.989542    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 11:42:24.060790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.060757    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 11:42:24.061813 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.061792    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" containerName="istio-proxy"
Oct 30 11:42:24.065119 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.065100    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.092621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.092599    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 11:42:24.107576 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.107534    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.107707 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.107667    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7ms\" (UniqueName: \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.209781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.209688    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vpm\" (UniqueName: \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.210114 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.210081    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7ms\" (UniqueName: \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.210208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.210186    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.210276 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.210253    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.211216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.211194    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.239587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.239553    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7ms\" (UniqueName: \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\") pod \"payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.273157 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.273133    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.311736 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.311699    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.311869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.311788    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vpm\" (UniqueName: \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.312435 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.312415    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.333657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.333633    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vpm\" (UniqueName: \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\") pod \"payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.378392 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.378358    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.567068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.567039    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 11:42:24.569114 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:24.569093    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d146c4f_3052_4e60_8dbd_e44c4463decc.slice/crio-4a34b2682ca1620ea06d88c6b9077ad2507cf9a22941c48949f11a7798387f98 WatchSource:0}: Error finding container 4a34b2682ca1620ea06d88c6b9077ad2507cf9a22941c48949f11a7798387f98: Status 404 returned error can't find the container with id 4a34b2682ca1620ea06d88c6b9077ad2507cf9a22941c48949f11a7798387f98
Oct 30 11:42:24.674661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.674630    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 11:42:24.676914 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:24.676885    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987b63ab_2980_4e31_934a_b9c790783b19.slice/crio-8ac781d7edd7c28c876c44130af9a25385ac9d167f1b183b8694fa86dfad278b WatchSource:0}: Error finding container 8ac781d7edd7c28c876c44130af9a25385ac9d167f1b183b8694fa86dfad278b: Status 404 returned error can't find the container with id 8ac781d7edd7c28c876c44130af9a25385ac9d167f1b183b8694fa86dfad278b
Oct 30 11:42:24.912083 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.912048    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a812aa-b130-4137-93ef-6ec5f21f1ff3" path="/var/lib/kubelet/pods/23a812aa-b130-4137-93ef-6ec5f21f1ff3/volumes"
Oct 30 11:42:24.938699 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.938658    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" event={"ID":"4d146c4f-3052-4e60-8dbd-e44c4463decc","Type":"ContainerStarted","Data":"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"}
Oct 30 11:42:24.938874 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.938727    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" event={"ID":"4d146c4f-3052-4e60-8dbd-e44c4463decc","Type":"ContainerStarted","Data":"4a34b2682ca1620ea06d88c6b9077ad2507cf9a22941c48949f11a7798387f98"}
Oct 30 11:42:24.938874 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.938827    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:24.941078 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.941045    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" event={"ID":"987b63ab-2980-4e31-934a-b9c790783b19","Type":"ContainerStarted","Data":"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"}
Oct 30 11:42:24.941244 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.941114    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" event={"ID":"987b63ab-2980-4e31-934a-b9c790783b19","Type":"ContainerStarted","Data":"8ac781d7edd7c28c876c44130af9a25385ac9d167f1b183b8694fa86dfad278b"}
Oct 30 11:42:24.941244 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.941239    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:24.974389 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.974323    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" podStartSLOduration=1.9743104900000001 podStartE2EDuration="1.97431049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:24.973647277 +0000 UTC m=+744.193932186" watchObservedRunningTime="2026-10-30 11:42:24.97431049 +0000 UTC m=+744.194595406"
Oct 30 11:42:24.977653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:24.977613    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 11:42:25.014967 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:25.014921    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" podStartSLOduration=1.014908317 podStartE2EDuration="1.014908317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:25.013947402 +0000 UTC m=+744.234232311" watchObservedRunningTime="2026-10-30 11:42:25.014908317 +0000 UTC m=+744.235193228"
Oct 30 11:42:27.784685 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.784653    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 11:42:27.790244 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.790221    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:27.806771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.806751    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-4e261b-serving-cert\""
Oct 30 11:42:27.814808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.814784    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 11:42:27.942024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.941990    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:27.942024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.942030    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:27.942269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.942052    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jr7\" (UniqueName: \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:27.942269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:27.942069    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.043421 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.043326    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.043421 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.043385    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.043626 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.043433    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jr7\" (UniqueName: \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.043626 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.043464    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.044175 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.044147    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.045931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.045907    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.045931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.045920    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.066618 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.066588    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jr7\" (UniqueName: \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\") pod \"maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.102891 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.102863    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.159737 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.159693    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:42:28.160013 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.159984    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy" containerID="cri-o://49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa" gracePeriod=30
Oct 30 11:42:28.378731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.378697    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:42:28.411191 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.411160    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:42:28.430947 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.430923    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 11:42:28.433157 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:28.433135    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23938af_54cd_4b54_b1b2_86057cc83721.slice/crio-043574b34e2c3bb559a7b9f132cbe51e5a4d74c37fc4377ecab2f9f2d5000568 WatchSource:0}: Error finding container 043574b34e2c3bb559a7b9f132cbe51e5a4d74c37fc4377ecab2f9f2d5000568: Status 404 returned error can't find the container with id 043574b34e2c3bb559a7b9f132cbe51e5a4d74c37fc4377ecab2f9f2d5000568
Oct 30 11:42:28.962130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.962092    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" event={"ID":"c23938af-54cd-4b54-b1b2-86057cc83721","Type":"ContainerStarted","Data":"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"}
Oct 30 11:42:28.962616 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.962148    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" event={"ID":"c23938af-54cd-4b54-b1b2-86057cc83721","Type":"ContainerStarted","Data":"043574b34e2c3bb559a7b9f132cbe51e5a4d74c37fc4377ecab2f9f2d5000568"}
Oct 30 11:42:28.962616 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.962241    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:28.999968 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:28.999894    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" podStartSLOduration=1.6992343619999999 podStartE2EDuration="1.999882174s" totalImagesPullingTime="300.647812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:28.998925769 +0000 UTC m=+748.219210680" watchObservedRunningTime="2026-10-30 11:42:28.999882174 +0000 UTC m=+748.220167084"
Oct 30 11:42:29.683342 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.683282    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:29.683536 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.683503    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" podUID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" containerName="maas-api" containerID="cri-o://d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0" gracePeriod=30
Oct 30 11:42:29.939852 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.939790    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:29.968932 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.968900    4453 generic.go:356] "Generic (PLEG): container finished" podID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" containerID="d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0" exitCode=0
Oct 30 11:42:29.969365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.968944    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"
Oct 30 11:42:29.969365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.968973    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" event={"ID":"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a","Type":"ContainerDied","Data":"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"}
Oct 30 11:42:29.969365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.969021    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw" event={"ID":"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a","Type":"ContainerDied","Data":"e40addc51b16337909d6cb205022242a7b05165662f920da38bac78a2b3af060"}
Oct 30 11:42:29.969365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.969043    4453 scope.go:122] "RemoveContainer" containerID="d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"
Oct 30 11:42:29.978434 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.978310    4453 scope.go:122] "RemoveContainer" containerID="d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"
Oct 30 11:42:29.978636 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:29.978617    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0\": container with ID starting with d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0 not found: ID does not exist" containerID="d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"
Oct 30 11:42:29.978711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:29.978645    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0"} err="failed to get container status \"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0\": rpc error: code = NotFound desc = could not find container \"d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0\": container with ID starting with d2d91e5bdeaf97bfba0d918a629295ca0d66a51aa2a95ac18bd9d6ea3a2556f0 not found: ID does not exist"
Oct 30 11:42:30.065030 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.064999    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\") pod \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") "
Oct 30 11:42:30.065180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.065084    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\" (UniqueName: \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\") pod \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") "
Oct 30 11:42:30.065180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.065116    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\") pod \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") "
Oct 30 11:42:30.065315 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.065225    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\") pod \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\" (UID: \"b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a\") "
Oct 30 11:42:30.065430 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.065402    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca" pod "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" (UID: "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:30.065604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.065584    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:30.067411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.067373    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls" pod "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" (UID: "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:30.067488 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.067447    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584" pod "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" (UID: "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a"). InnerVolumeSpecName "kube-api-access-2n584". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:30.067488 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.067445    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs" pod "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" (UID: "b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:30.166219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.166180    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n584\" (UniqueName: \"kubernetes.io/projected/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-kube-api-access-2n584\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:30.166219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.166208    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:30.166219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.166221    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:30.308176 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.308140    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:30.312182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.312157    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90364398-965944996-6kptw"]
Oct 30 11:42:30.910914 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:30.910884    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" path="/var/lib/kubelet/pods/b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a/volumes"
Oct 30 11:42:31.756353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.755847    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:42:31.769355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.768501    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" containerName="maas-api"
Oct 30 11:42:31.769355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.768530    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" containerName="maas-api"
Oct 30 11:42:31.769355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.768670    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98f8ae8-61a7-4b67-9701-d4a7ff9fdb1a" containerName="maas-api"
Oct 30 11:42:31.785353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.775743    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:42:31.785353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.775864    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.800174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.791291    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3ccdab-openshift-default-dockercfg-zcvfb\""
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886776    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886825    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886873    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886912    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886956    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.886997    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.887021    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts5f\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.887055    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.887356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.887096    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988014 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.987977    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988037    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988067    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988101    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988129    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988162    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988185    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts5f\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988216    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988260    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988432    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988558 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988507    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988753 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988591    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.988948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.988921    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.989132 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.989106    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.990931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.990905    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:31.991065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:31.990915    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:32.012061 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.011995    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts5f\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:32.020935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.020909    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\") pod \"e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") " pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:32.172725 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.172680    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:32.515938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.515912    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:42:32.518070 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:32.518042    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26873a_a86e_4153_a051_1f658c8650ad.slice/crio-7cd56a320d12842484e3c6cd94dd0aef4b821896e77dc708a7e8ef552aea8888 WatchSource:0}: Error finding container 7cd56a320d12842484e3c6cd94dd0aef4b821896e77dc708a7e8ef552aea8888: Status 404 returned error can't find the container with id 7cd56a320d12842484e3c6cd94dd0aef4b821896e77dc708a7e8ef552aea8888
Oct 30 11:42:32.520384 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.520348    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:32.520453 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.520415    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:32.520453 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.520445    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:32.985662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.985628    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerStarted","Data":"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"}
Oct 30 11:42:32.986038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:32.985676    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerStarted","Data":"7cd56a320d12842484e3c6cd94dd0aef4b821896e77dc708a7e8ef552aea8888"}
Oct 30 11:42:33.026192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.026145    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podStartSLOduration=2.02613377 podStartE2EDuration="2.02613377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:33.024857409 +0000 UTC m=+752.245142319" watchObservedRunningTime="2026-10-30 11:42:33.02613377 +0000 UTC m=+752.246418680"
Oct 30 11:42:33.174054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.174018    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:33.179550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.179522    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:33.219011 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:33.218980    4453 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20c9da5_9c97_4bdf_a7cf_13dfeee18eb8.slice/crio-conmon-49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 11:42:33.379161 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.379124    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" podUID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" containerName="payload-processing" containerID="cri-o://2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a" gracePeriod=25
Oct 30 11:42:33.411585 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.411549    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" podUID="422215a3-12eb-4976-8e62-d6c8a2856c31" containerName="payload-pre-processing" containerID="cri-o://44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6" gracePeriod=25
Oct 30 11:42:33.524147 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.524123    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.604890    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.604971    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605023    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605074    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605104    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605147    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605622 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605202    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605622 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605236    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.605622 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.605348    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\") pod \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\" (UID: \"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8\") "
Oct 30 11:42:33.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.606130    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:33.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.606168    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:33.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.606356    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:33.606648 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.606462    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:33.607254 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.607225    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:33.609284 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.609237    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "kube-api-access-drxg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:33.611055 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.611004    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:33.611164 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.611135    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:42:33.611408 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.611373    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy" pod "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" (UID: "d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:42:33.674966 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.674938    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:42:33.682457 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.682436    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:42:33.706416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706388    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706413    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706425    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706439    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706452    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxg4\" (UniqueName: \"kubernetes.io/projected/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-kube-api-access-drxg4\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706465    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706477    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706489    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.706557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.706500    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.807873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.807797    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\") pod \"422215a3-12eb-4976-8e62-d6c8a2856c31\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") "
Oct 30 11:42:33.807873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.807847    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\") pod \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") "
Oct 30 11:42:33.808068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.807880    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\" (UniqueName: \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\") pod \"422215a3-12eb-4976-8e62-d6c8a2856c31\" (UID: \"422215a3-12eb-4976-8e62-d6c8a2856c31\") "
Oct 30 11:42:33.808068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.807942    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\" (UniqueName: \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\") pod \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\" (UID: \"23545519-1f4c-44c8-bd0f-7a4dae434f5a\") "
Oct 30 11:42:33.808245 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.808218    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume" pod "422215a3-12eb-4976-8e62-d6c8a2856c31" (UID: "422215a3-12eb-4976-8e62-d6c8a2856c31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:33.808298 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.808278    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume" pod "23545519-1f4c-44c8-bd0f-7a4dae434f5a" (UID: "23545519-1f4c-44c8-bd0f-7a4dae434f5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:33.810024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.810000    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9" pod "422215a3-12eb-4976-8e62-d6c8a2856c31" (UID: "422215a3-12eb-4976-8e62-d6c8a2856c31"). InnerVolumeSpecName "kube-api-access-zdzk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:33.810122 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.810066    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr" pod "23545519-1f4c-44c8-bd0f-7a4dae434f5a" (UID: "23545519-1f4c-44c8-bd0f-7a4dae434f5a"). InnerVolumeSpecName "kube-api-access-ch8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:33.909145 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.909118    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzk9\" (UniqueName: \"kubernetes.io/projected/422215a3-12eb-4976-8e62-d6c8a2856c31-kube-api-access-zdzk9\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.909145 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.909143    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8kr\" (UniqueName: \"kubernetes.io/projected/23545519-1f4c-44c8-bd0f-7a4dae434f5a-kube-api-access-ch8kr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.909306 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.909153    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422215a3-12eb-4976-8e62-d6c8a2856c31-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.909306 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.909163    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23545519-1f4c-44c8-bd0f-7a4dae434f5a-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:33.991949 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.991921    4453 generic.go:356] "Generic (PLEG): container finished" podID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerID="49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa" exitCode=0
Oct 30 11:42:33.992380 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.991984    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"
Oct 30 11:42:33.992380 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.992007    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerDied","Data":"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"}
Oct 30 11:42:33.992380 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.992061    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp" event={"ID":"d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8","Type":"ContainerDied","Data":"ba21b692615621eaf3d6bbbd13b8f0c766accc84d82d4f566ea94780314d7c44"}
Oct 30 11:42:33.992380 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.992084    4453 scope.go:122] "RemoveContainer" containerID="49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"
Oct 30 11:42:33.993706 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.993683    4453 generic.go:356] "Generic (PLEG): container finished" podID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" containerID="2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a" exitCode=0
Oct 30 11:42:33.993808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.993724    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"
Oct 30 11:42:33.993808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.993768    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" event={"ID":"23545519-1f4c-44c8-bd0f-7a4dae434f5a","Type":"ContainerDied","Data":"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"}
Oct 30 11:42:33.993808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.993804    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd" event={"ID":"23545519-1f4c-44c8-bd0f-7a4dae434f5a","Type":"ContainerDied","Data":"ff9db0e1a76334c94f6260839d24dccc03e6951925a374038ee72b8494c97b11"}
Oct 30 11:42:33.995298 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.995277    4453 generic.go:356] "Generic (PLEG): container finished" podID="422215a3-12eb-4976-8e62-d6c8a2856c31" containerID="44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6" exitCode=0
Oct 30 11:42:33.995404 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.995305    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" event={"ID":"422215a3-12eb-4976-8e62-d6c8a2856c31","Type":"ContainerDied","Data":"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"}
Oct 30 11:42:33.995404 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.995362    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n" event={"ID":"422215a3-12eb-4976-8e62-d6c8a2856c31","Type":"ContainerDied","Data":"1831d5bec53bcf4997286d1d5bdbe1d1abe54cb744a92ee59867d57be779fda4"}
Oct 30 11:42:33.995404 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.995373    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"
Oct 30 11:42:33.995608 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.995588    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:33.996825 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:33.996807    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:34.015128 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.015011    4453 scope.go:122] "RemoveContainer" containerID="7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"
Oct 30 11:42:34.023022 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.023008    4453 scope.go:122] "RemoveContainer" containerID="49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"
Oct 30 11:42:34.023257 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:34.023241    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa\": container with ID starting with 49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa not found: ID does not exist" containerID="49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"
Oct 30 11:42:34.023313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.023263    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa"} err="failed to get container status \"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa\": rpc error: code = NotFound desc = could not find container \"49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa\": container with ID starting with 49f157f246e8c6f3b201db9b576674b4a40489f64ecdaf1397ffd5d4574daffa not found: ID does not exist"
Oct 30 11:42:34.023313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.023278    4453 scope.go:122] "RemoveContainer" containerID="7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"
Oct 30 11:42:34.023524 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:34.023509    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd\": container with ID starting with 7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd not found: ID does not exist" containerID="7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"
Oct 30 11:42:34.023574 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.023528    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd"} err="failed to get container status \"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd\": rpc error: code = NotFound desc = could not find container \"7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd\": container with ID starting with 7d6820aa20d89af2438853472521b263e04f30a953ddf4ce2c133f998d170fbd not found: ID does not exist"
Oct 30 11:42:34.023574 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.023541    4453 scope.go:122] "RemoveContainer" containerID="2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"
Oct 30 11:42:34.031247 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.031229    4453 scope.go:122] "RemoveContainer" containerID="2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"
Oct 30 11:42:34.031513 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:34.031495    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a\": container with ID starting with 2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a not found: ID does not exist" containerID="2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"
Oct 30 11:42:34.031569 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.031520    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a"} err="failed to get container status \"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a\": rpc error: code = NotFound desc = could not find container \"2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a\": container with ID starting with 2ab44aa236759f0b7563ba62b7677d72756ca7e7e10305375136b886c10abf2a not found: ID does not exist"
Oct 30 11:42:34.031569 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.031536    4453 scope.go:122] "RemoveContainer" containerID="44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"
Oct 30 11:42:34.038555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.038539    4453 scope.go:122] "RemoveContainer" containerID="44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"
Oct 30 11:42:34.038782 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:34.038763    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6\": container with ID starting with 44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6 not found: ID does not exist" containerID="44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"
Oct 30 11:42:34.038829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.038789    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6"} err="failed to get container status \"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6\": rpc error: code = NotFound desc = could not find container \"44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6\": container with ID starting with 44ecf01735aa6c3ca8ad68202f31789ea8cdba655d6e4af3e2b5aff8aa49c5c6 not found: ID does not exist"
Oct 30 11:42:34.107935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.107910    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:42:34.112991 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.112361    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0836c883-openshift-default-56665d7c5-hthqp"]
Oct 30 11:42:34.141766 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.141724    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:42:34.152931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.152730    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-90364398-77778cb55d-d7kzd"]
Oct 30 11:42:34.175125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.175098    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:42:34.178919 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.178896    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90364398-5454d7d8f9-5qv2n"]
Oct 30 11:42:34.909762 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.909733    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" path="/var/lib/kubelet/pods/23545519-1f4c-44c8-bd0f-7a4dae434f5a/volumes"
Oct 30 11:42:34.910090 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.910077    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422215a3-12eb-4976-8e62-d6c8a2856c31" path="/var/lib/kubelet/pods/422215a3-12eb-4976-8e62-d6c8a2856c31/volumes"
Oct 30 11:42:34.910503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.910490    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" path="/var/lib/kubelet/pods/d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8/volumes"
Oct 30 11:42:34.974700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:34.974671    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 11:42:35.239193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239101    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 11:42:35.239711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239678    4453 cpu_manager.go:414] "containerMap: removing container" podUID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" containerName="payload-processing"
Oct 30 11:42:35.239711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239703    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" containerName="payload-processing"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239738    4453 cpu_manager.go:414] "containerMap: removing container" podUID="422215a3-12eb-4976-8e62-d6c8a2856c31" containerName="payload-pre-processing"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239745    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422215a3-12eb-4976-8e62-d6c8a2856c31" containerName="payload-pre-processing"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239766    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239774    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239784    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239792    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239869    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="422215a3-12eb-4976-8e62-d6c8a2856c31" containerName="payload-pre-processing"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239878    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239885    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20c9da5-9c97-4bdf-a7cf-13dfeee18eb8" containerName="istio-proxy"
Oct 30 11:42:35.239889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.239893    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="23545519-1f4c-44c8-bd0f-7a4dae434f5a" containerName="payload-processing"
Oct 30 11:42:35.244594 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.244578    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.261771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.261749    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-849669-dockercfg-ppdcg\""
Oct 30 11:42:35.261771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.261749    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-849669\""
Oct 30 11:42:35.266678 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.266659    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 11:42:35.294697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.294667    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 11:42:35.320352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.299045    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.326088 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.321239    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 11:42:35.425975 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.425936    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.425975 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.425977    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.426171 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.426090    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9bd\" (UniqueName: \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.426227 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.426207    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.527410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.527317    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9bd\" (UniqueName: \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.527577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.527430    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.527577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.527509    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.527577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.527534    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.528395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.528361    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.528395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.528389    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.555462 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.555439    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\") pod \"payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.555565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.555492    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9bd\" (UniqueName: \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\") pod \"payload-processing-e2e-worker-w4-849669-f9698874b-mvl59\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.557197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.557179    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:35.619915 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.619885    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:35.858047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.858021    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 11:42:35.860209 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:35.860181    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb44dea0_a97c_4fc7_860d_60d44d72d724.slice/crio-f5818ce47c1e28d22578f5079ae285510dba88296d9678fab20a7c8fe91caecf WatchSource:0}: Error finding container f5818ce47c1e28d22578f5079ae285510dba88296d9678fab20a7c8fe91caecf: Status 404 returned error can't find the container with id f5818ce47c1e28d22578f5079ae285510dba88296d9678fab20a7c8fe91caecf
Oct 30 11:42:35.927657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.927631    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 11:42:35.930589 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:35.930561    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac8816f_b9fd_4e11_99df_3314b405dcb7.slice/crio-4750e6e052e878881e21d13767565fbedb3980e6c3db947d263f877f8055393f WatchSource:0}: Error finding container 4750e6e052e878881e21d13767565fbedb3980e6c3db947d263f877f8055393f: Status 404 returned error can't find the container with id 4750e6e052e878881e21d13767565fbedb3980e6c3db947d263f877f8055393f
Oct 30 11:42:35.949600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.949577    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 11:42:35.949689 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:35.949645    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 11:42:36.014542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.014500    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" event={"ID":"fb44dea0-a97c-4fc7-860d-60d44d72d724","Type":"ContainerStarted","Data":"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"}
Oct 30 11:42:36.014711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.014565    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:36.014711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.014588    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" event={"ID":"fb44dea0-a97c-4fc7-860d-60d44d72d724","Type":"ContainerStarted","Data":"f5818ce47c1e28d22578f5079ae285510dba88296d9678fab20a7c8fe91caecf"}
Oct 30 11:42:36.016973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.016949    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" event={"ID":"cac8816f-b9fd-4e11-99df-3314b405dcb7","Type":"ContainerStarted","Data":"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"}
Oct 30 11:42:36.017082 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.016986    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" event={"ID":"cac8816f-b9fd-4e11-99df-3314b405dcb7","Type":"ContainerStarted","Data":"4750e6e052e878881e21d13767565fbedb3980e6c3db947d263f877f8055393f"}
Oct 30 11:42:36.017082 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.017015    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:36.074462 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.074396    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" podStartSLOduration=1.074315203 podStartE2EDuration="1.074315203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:36.073484176 +0000 UTC m=+755.293769113" watchObservedRunningTime="2026-10-30 11:42:36.074315203 +0000 UTC m=+755.294600112"
Oct 30 11:42:36.116441 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:36.116242    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" podStartSLOduration=1.116224679 podStartE2EDuration="1.116224679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:36.113761177 +0000 UTC m=+755.334046088" watchObservedRunningTime="2026-10-30 11:42:36.116224679 +0000 UTC m=+755.336509590"
Oct 30 11:42:37.569582 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:37.569553    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:42:39.122782 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.122745    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 11:42:39.126812 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.126791    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.139167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.139144    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-1d9b1a\""
Oct 30 11:42:39.150556 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.150526    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 11:42:39.196411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.196381    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 11:42:39.200042 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.200022    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.216108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.216088    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-1d9b1a-dockercfg-rxkrr\""
Oct 30 11:42:39.221427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.221403    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 11:42:39.261291 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.261253    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.261508 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.261487    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwft\" (UniqueName: \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.279088 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.279042    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 11:42:39.283577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.283549    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.303911 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.303870    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 11:42:39.323424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.323404    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-849669-serving-cert\""
Oct 30 11:42:39.362715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.362687    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.362715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.362718    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpnk\" (UniqueName: \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.362888 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.362751    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwft\" (UniqueName: \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.362888 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.362872    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.363545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.363529    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.383225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.383170    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwft\" (UniqueName: \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\") pod \"payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.440871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.440839    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:39.464484 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464456    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56wx\" (UniqueName: \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.464606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464534    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.464606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464586    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.464677 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464614    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.464677 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464630    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpnk\" (UniqueName: \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.464677 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.464647    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.465316 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.465298    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.486215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.486185    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpnk\" (UniqueName: \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\") pod \"payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.513129 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.513106    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:39.565904 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.565403    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.565904 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.565455    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.565904 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.565487    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.565904 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.565552    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56wx\" (UniqueName: \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.566309 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.566286    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.568562 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.568035    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.568562 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.568124    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.589318 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.587402    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56wx\" (UniqueName: \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\") pod \"maas-api-e2e-worker-w4-849669-75c786945d-k9xff\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.598058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.598037    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:39.741028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.740993    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 11:42:39.743180 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:39.743151    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dc9bf9_0842_464f_a70c_e1328f61764f.slice/crio-2acbb6d6349990f12e04105913e093c37ccb264d070642ba7ca645b9c1878b78 WatchSource:0}: Error finding container 2acbb6d6349990f12e04105913e093c37ccb264d070642ba7ca645b9c1878b78: Status 404 returned error can't find the container with id 2acbb6d6349990f12e04105913e093c37ccb264d070642ba7ca645b9c1878b78
Oct 30 11:42:39.825029 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.825005    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 11:42:39.826920 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:39.826890    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72e6d7c_cd0f_4e7b_89a3_7894d03f2274.slice/crio-e9b68286a6386dd7517ea5a4c20d857b91309936f0e9369e0defdc85e59bf44b WatchSource:0}: Error finding container e9b68286a6386dd7517ea5a4c20d857b91309936f0e9369e0defdc85e59bf44b: Status 404 returned error can't find the container with id e9b68286a6386dd7517ea5a4c20d857b91309936f0e9369e0defdc85e59bf44b
Oct 30 11:42:39.899054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:39.898991    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 11:42:39.900725 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:39.900672    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f08044_0f0d_47ce_8640_b4593e1ee5c1.slice/crio-65658e3359cbdfd865aeff384a5a9e1806e01ce89032f2f881f87cfd175dfb1b WatchSource:0}: Error finding container 65658e3359cbdfd865aeff384a5a9e1806e01ce89032f2f881f87cfd175dfb1b: Status 404 returned error can't find the container with id 65658e3359cbdfd865aeff384a5a9e1806e01ce89032f2f881f87cfd175dfb1b
Oct 30 11:42:40.038853 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.038819    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" event={"ID":"69f08044-0f0d-47ce-8640-b4593e1ee5c1","Type":"ContainerStarted","Data":"65658e3359cbdfd865aeff384a5a9e1806e01ce89032f2f881f87cfd175dfb1b"}
Oct 30 11:42:40.040466 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.040434    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" event={"ID":"44dc9bf9-0842-464f-a70c-e1328f61764f","Type":"ContainerStarted","Data":"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"}
Oct 30 11:42:40.040623 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.040484    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" event={"ID":"44dc9bf9-0842-464f-a70c-e1328f61764f","Type":"ContainerStarted","Data":"2acbb6d6349990f12e04105913e093c37ccb264d070642ba7ca645b9c1878b78"}
Oct 30 11:42:40.040623 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.040599    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:40.041905 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.041873    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" event={"ID":"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274","Type":"ContainerStarted","Data":"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"}
Oct 30 11:42:40.041994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.041915    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" event={"ID":"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274","Type":"ContainerStarted","Data":"e9b68286a6386dd7517ea5a4c20d857b91309936f0e9369e0defdc85e59bf44b"}
Oct 30 11:42:40.041994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.041955    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:40.076906 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.076863    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" podStartSLOduration=1.076851695 podStartE2EDuration="1.076851695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:40.076216881 +0000 UTC m=+759.296501791" watchObservedRunningTime="2026-10-30 11:42:40.076851695 +0000 UTC m=+759.297136635"
Oct 30 11:42:40.111768 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:40.111732    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" podStartSLOduration=1.111718021 podStartE2EDuration="1.111718021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:40.110751724 +0000 UTC m=+759.331036634" watchObservedRunningTime="2026-10-30 11:42:40.111718021 +0000 UTC m=+759.332002932"
Oct 30 11:42:41.002060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.002019    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:42:41.002509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.002245    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" podUID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" containerName="maas-api" containerID="cri-o://4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b" gracePeriod=30
Oct 30 11:42:41.049130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.049016    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" event={"ID":"69f08044-0f0d-47ce-8640-b4593e1ee5c1","Type":"ContainerStarted","Data":"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"}
Oct 30 11:42:41.049130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.049095    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:41.095090 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.095045    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" podStartSLOduration=1.779747741 podStartE2EDuration="2.09502936s" totalImagesPullingTime="315.281619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:41.094691066 +0000 UTC m=+760.314975977" watchObservedRunningTime="2026-10-30 11:42:41.09502936 +0000 UTC m=+760.315314270"
Oct 30 11:42:41.265196 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.265124    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:41.383108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383074    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\") pod \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") "
Oct 30 11:42:41.383260 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383243    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\") pod \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") "
Oct 30 11:42:41.383300 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383283    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\") pod \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") "
Oct 30 11:42:41.383393 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383360    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\" (UniqueName: \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\") pod \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\" (UID: \"5896fc64-ade9-4326-9eed-caf0ca3b52f0\") "
Oct 30 11:42:41.383531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383500    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca" pod "5896fc64-ade9-4326-9eed-caf0ca3b52f0" (UID: "5896fc64-ade9-4326-9eed-caf0ca3b52f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:42:41.383742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.383723    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5896fc64-ade9-4326-9eed-caf0ca3b52f0-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:41.385303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.385277    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc" pod "5896fc64-ade9-4326-9eed-caf0ca3b52f0" (UID: "5896fc64-ade9-4326-9eed-caf0ca3b52f0"). InnerVolumeSpecName "kube-api-access-5r4mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:42:41.385415 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.385301    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls" pod "5896fc64-ade9-4326-9eed-caf0ca3b52f0" (UID: "5896fc64-ade9-4326-9eed-caf0ca3b52f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:41.385415 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.385315    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs" pod "5896fc64-ade9-4326-9eed-caf0ca3b52f0" (UID: "5896fc64-ade9-4326-9eed-caf0ca3b52f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:42:41.484624 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.484585    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:41.484624 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.484619    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5896fc64-ade9-4326-9eed-caf0ca3b52f0-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:41.484624 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.484629    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4mc\" (UniqueName: \"kubernetes.io/projected/5896fc64-ade9-4326-9eed-caf0ca3b52f0-kube-api-access-5r4mc\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:42:41.696596 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.696563    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 11:42:41.697149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.697126    4453 cpu_manager.go:414] "containerMap: removing container" podUID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" containerName="maas-api"
Oct 30 11:42:41.697149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.697150    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" containerName="maas-api"
Oct 30 11:42:41.697322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.697246    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" containerName="maas-api"
Oct 30 11:42:41.700699 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.700678    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.729349 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.728518    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3ccdab-serving-cert\""
Oct 30 11:42:41.736040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.736007    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 11:42:41.888907 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.888865    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7p6\" (UniqueName: \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.889055 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.889008    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.889195 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.889168    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.889234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.889220    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.989938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.989860    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.990089 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.989957    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.990089 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.990051    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.990208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.990127    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7p6\" (UniqueName: \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.990715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.990688    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.992442 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.992425    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:41.992866 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:41.992850    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:42.011290 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.011268    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7p6\" (UniqueName: \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\") pod \"maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:42.015111 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.015090    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:42.056029 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.055998    4453 generic.go:356] "Generic (PLEG): container finished" podID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" containerID="4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b" exitCode=0
Oct 30 11:42:42.056198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.056054    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"
Oct 30 11:42:42.056198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.056071    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" event={"ID":"5896fc64-ade9-4326-9eed-caf0ca3b52f0","Type":"ContainerDied","Data":"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"}
Oct 30 11:42:42.056198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.056125    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq" event={"ID":"5896fc64-ade9-4326-9eed-caf0ca3b52f0","Type":"ContainerDied","Data":"d9ac046b92492543e4c8e5b7d36b060d9c8108550ad9f70e51212099c7a36547"}
Oct 30 11:42:42.056198 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.056147    4453 scope.go:122] "RemoveContainer" containerID="4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"
Oct 30 11:42:42.067384 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.067262    4453 scope.go:122] "RemoveContainer" containerID="4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"
Oct 30 11:42:42.067650 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:42:42.067588    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b\": container with ID starting with 4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b not found: ID does not exist" containerID="4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"
Oct 30 11:42:42.067650 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.067619    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b"} err="failed to get container status \"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b\": rpc error: code = NotFound desc = could not find container \"4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b\": container with ID starting with 4ebc1decac1ef448a82370e116d13c039ac625f5150d8e507a74b767ed59232b not found: ID does not exist"
Oct 30 11:42:42.123518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.123472    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:42:42.126019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.125981    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64715175-958c45dbc-5qxhq"]
Oct 30 11:42:42.314496 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.314469    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 11:42:42.316040 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:42.316013    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bffa8b_cc2a_4829_9615_b92ba71052c1.slice/crio-87131a3b6eb77020a53c1f14f58d537ccfc173b0488fae2b10804c84656293ca WatchSource:0}: Error finding container 87131a3b6eb77020a53c1f14f58d537ccfc173b0488fae2b10804c84656293ca: Status 404 returned error can't find the container with id 87131a3b6eb77020a53c1f14f58d537ccfc173b0488fae2b10804c84656293ca
Oct 30 11:42:42.910843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:42.910809    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5896fc64-ade9-4326-9eed-caf0ca3b52f0" path="/var/lib/kubelet/pods/5896fc64-ade9-4326-9eed-caf0ca3b52f0/volumes"
Oct 30 11:42:43.063582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.063543    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" event={"ID":"a3bffa8b-cc2a-4829-9615-b92ba71052c1","Type":"ContainerStarted","Data":"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"}
Oct 30 11:42:43.063970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.063606    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" event={"ID":"a3bffa8b-cc2a-4829-9615-b92ba71052c1","Type":"ContainerStarted","Data":"87131a3b6eb77020a53c1f14f58d537ccfc173b0488fae2b10804c84656293ca"}
Oct 30 11:42:43.063970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.063642    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:43.105648 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.105582    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" podStartSLOduration=1.83053661 podStartE2EDuration="2.105561674s" totalImagesPullingTime="275.025064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:43.104016628 +0000 UTC m=+762.324301540" watchObservedRunningTime="2026-10-30 11:42:43.105561674 +0000 UTC m=+762.325846588"
Oct 30 11:42:43.911618 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.911529    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 11:42:43.916346 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.916314    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:43.940047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.940021    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-1d9b1a-serving-cert\""
Oct 30 11:42:43.946923 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:43.946745    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 11:42:44.010058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.010019    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.010224 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.010080    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.010224 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.010199    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.010312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.010290    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54rl\" (UniqueName: \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.111019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.110977    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.111511 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.111034    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.111511 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.111076    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.111511 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.111166    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54rl\" (UniqueName: \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.111810 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.111746    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.117267 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.114026    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.117267 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.114064    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.134310 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.134276    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54rl\" (UniqueName: \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\") pod \"maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.232073 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.231981    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:44.534935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:44.534908    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 11:42:44.536900 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:42:44.536873    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584d9ada_9105_47cd_b5a1_6ee303a7b3cf.slice/crio-13dd9a520d7db03c0ae0d746627bfedc795de1d5909beacdc5a2c912b5a82ebe WatchSource:0}: Error finding container 13dd9a520d7db03c0ae0d746627bfedc795de1d5909beacdc5a2c912b5a82ebe: Status 404 returned error can't find the container with id 13dd9a520d7db03c0ae0d746627bfedc795de1d5909beacdc5a2c912b5a82ebe
Oct 30 11:42:45.077398 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:45.077360    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" event={"ID":"584d9ada-9105-47cd-b5a1-6ee303a7b3cf","Type":"ContainerStarted","Data":"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"}
Oct 30 11:42:45.077595 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:45.077423    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" event={"ID":"584d9ada-9105-47cd-b5a1-6ee303a7b3cf","Type":"ContainerStarted","Data":"13dd9a520d7db03c0ae0d746627bfedc795de1d5909beacdc5a2c912b5a82ebe"}
Oct 30 11:42:45.077595 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:45.077480    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:45.122578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:45.122527    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" podStartSLOduration=1.738605105 podStartE2EDuration="2.122512938s" totalImagesPullingTime="383.907833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:42:45.11915399 +0000 UTC m=+764.339438902" watchObservedRunningTime="2026-10-30 11:42:45.122512938 +0000 UTC m=+764.342797848"
Oct 30 11:42:47.023881 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:47.023847    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 11:42:47.024259 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:47.023912    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 11:42:47.062941 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:47.062913    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 11:42:49.076110 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:49.076080    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 11:42:51.051699 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:51.051668    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 11:42:51.052141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:51.051739    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 11:42:51.102538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:51.102510    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 11:42:57.177134 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.177088    4453 generic.go:356] "Generic (PLEG): container finished" podID="6e26873a-a86e-4153-a051-1f658c8650ad" containerID="dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9" exitCode=0
Oct 30 11:42:57.177621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.177163    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerDied","Data":"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"}
Oct 30 11:42:57.177683 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.177668    4453 scope.go:122] "RemoveContainer" containerID="dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"
Oct 30 11:42:57.179243 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.179199    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:57.179349 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.179291    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:57.179419 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:57.179352    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:42:58.184237 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:58.184204    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerStarted","Data":"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"}
Oct 30 11:42:59.173138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:59.173105    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:42:59.173369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:59.173218    4453 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 11:42:59.173369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:42:59.173271    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 11:43:00.174037 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:00.174003    4453 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 11:43:00.174462 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:00.174060    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 11:43:01.173227 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:01.173190    4453 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 11:43:01.173424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:01.173261    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 11:43:02.173246 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:02.173211    4453 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 11:43:02.173671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:02.173269    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 11:43:03.173938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:03.173905    4453 patch_prober.go:29] interesting pod/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Oct 30 11:43:03.174319 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:03.173956    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Oct 30 11:43:03.996250 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:03.996214    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:43:04.178231 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:04.178201    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:43:04.238457 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:04.238426    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:43:06.472323 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.472291    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 11:43:06.483614 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.483584    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.509874 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.509838    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e43a66-serving-cert\""
Oct 30 11:43:06.512918 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.512894    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 11:43:06.599299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.599263    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.599485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.599390    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.599485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.599426    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmgn\" (UniqueName: \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.599485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.599472    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.701024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.700982    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.701203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.701076    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.701203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.701127    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.701203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.701146    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmgn\" (UniqueName: \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.701317 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:43:06.701263    4453 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-serving-cert: secret "maas-api-e2e-worker-w1-e43a66-serving-cert" not found
Oct 30 11:43:06.701388 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:43:06.701372    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls podName:818bca10-8dbd-4d59-98c6-4b60b95f0ae2 nodeName:}" failed. No retries permitted until 2026-10-30 11:43:07.201348269 +0000 UTC m=+786.421633158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls") pod "maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" (UID: "818bca10-8dbd-4d59-98c6-4b60b95f0ae2") : secret "maas-api-e2e-worker-w1-e43a66-serving-cert" not found
Oct 30 11:43:06.701820 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.701796    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.703569 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.703550    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:06.722702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:06.722629    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmgn\" (UniqueName: \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:07.205850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:07.205812    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:07.208247 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:07.208209    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:07.396829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:07.396774    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:07.569203 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:43:07.569117    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:43:07.698581 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:07.698535    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 11:43:07.701431 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:43:07.701395    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818bca10_8dbd_4d59_98c6_4b60b95f0ae2.slice/crio-e70ab6e15dd0d898c893112abed511ccb13125abf5407a5338d5aca96b103912 WatchSource:0}: Error finding container e70ab6e15dd0d898c893112abed511ccb13125abf5407a5338d5aca96b103912: Status 404 returned error can't find the container with id e70ab6e15dd0d898c893112abed511ccb13125abf5407a5338d5aca96b103912
Oct 30 11:43:08.261807 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:08.261775    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" event={"ID":"818bca10-8dbd-4d59-98c6-4b60b95f0ae2","Type":"ContainerStarted","Data":"70153e3d40ac3776408e8e80fbb752e685910f99c2a5c3811fe69d00c820180d"}
Oct 30 11:43:08.261998 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:08.261825    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" event={"ID":"818bca10-8dbd-4d59-98c6-4b60b95f0ae2","Type":"ContainerStarted","Data":"e70ab6e15dd0d898c893112abed511ccb13125abf5407a5338d5aca96b103912"}
Oct 30 11:43:08.261998 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:08.261903    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:14.273278 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:14.273245    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 11:43:14.313755 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:14.313703    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" podStartSLOduration=7.9686264829999995 podStartE2EDuration="8.313686686s" totalImagesPullingTime="345.060203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:08.302731017 +0000 UTC m=+787.523015928" watchObservedRunningTime="2026-10-30 11:43:14.313686686 +0000 UTC m=+793.533971596"
Oct 30 11:43:16.332983 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.332952    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 11:43:16.345957 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.345932    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.357646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.357619    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 11:43:16.365407 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.365376    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"default-dockercfg-l6pc6\""
Oct 30 11:43:16.365530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.365471    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"premium-w3-4e261b-kserve-self-signed-certs\""
Oct 30 11:43:16.416819 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.416794    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"kube-root-ca.crt\""
Oct 30 11:43:16.416992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.416945    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"openshift-service-ca.crt\""
Oct 30 11:43:16.507029 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507000    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.507160 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507037    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.507160 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507055    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.507160 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507076    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.507277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507216    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzvf\" (UniqueName: \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.507277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.507257    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.608619 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608591    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzvf\" (UniqueName: \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.608802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608651    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.608802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608682    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.608802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608705    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.608802 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608732    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.609009 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.608801    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.609150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.609119    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.609266 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.609127    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.609266 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.609169    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.611027 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.611005    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.611275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.611256    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.631843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.631824    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzvf\" (UniqueName: \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\") pod \"premium-w3-4e261b-kserve-7579bcf9ff-qjn2l\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") " pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.657866 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.657842    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:16.954060 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:16.953733    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 11:43:16.955825 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:43:16.955793    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ae6a366_7df5_47d6_84ef_e8a24f6d8ad1.slice/crio-f61b3e65635f0f3f882c0e59e53b7bf2860aaf179560bef40c86c44c10947cb5 WatchSource:0}: Error finding container f61b3e65635f0f3f882c0e59e53b7bf2860aaf179560bef40c86c44c10947cb5: Status 404 returned error can't find the container with id f61b3e65635f0f3f882c0e59e53b7bf2860aaf179560bef40c86c44c10947cb5
Oct 30 11:43:17.326016 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:17.325917    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" event={"ID":"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1","Type":"ContainerStarted","Data":"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"}
Oct 30 11:43:17.326016 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:17.325956    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" event={"ID":"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1","Type":"ContainerStarted","Data":"f61b3e65635f0f3f882c0e59e53b7bf2860aaf179560bef40c86c44c10947cb5"}
Oct 30 11:43:17.363614 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:17.363562    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podStartSLOduration=1.363547448 podStartE2EDuration="1.363547448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:17.363281289 +0000 UTC m=+796.583566198" watchObservedRunningTime="2026-10-30 11:43:17.363547448 +0000 UTC m=+796.583832360"
Oct 30 11:43:17.658025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:17.657986    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:26.658389 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:26.658357    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:26.670592 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:26.670559    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:27.386193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:27.386165    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:43:34.884594 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.884553    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 11:43:34.889614 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.889588    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.918965 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.918932    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"openshift-service-ca.crt\""
Oct 30 11:43:34.922498 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.922480    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"kube-root-ca.crt\""
Oct 30 11:43:34.922555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.922483    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"default-dockercfg-qhshj\""
Oct 30 11:43:34.922792 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.922779    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"premium-w0-3ccdab-kserve-self-signed-certs\""
Oct 30 11:43:34.942648 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.942621    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 11:43:34.997281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997246    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.997491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997306    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.997491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997403    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.997491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997434    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnr8k\" (UniqueName: \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.997491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997455    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:34.997655 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:34.997520    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098534 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098499    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098557    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098594    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098628    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnr8k\" (UniqueName: \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098663    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098782    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.098988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.098968    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.099068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.099020    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.099171 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.099150    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.100929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.100900    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.101094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.101076    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.119518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.119484    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnr8k\" (UniqueName: \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\") pod \"premium-w0-3ccdab-kserve-796777c4b4-9h4rn\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") " pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.202492 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.202402    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:35.519225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:35.519147    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 11:43:35.522528 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:43:35.522499    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8b26e16_f0aa_4d69_ab7e_6692ad20c506.slice/crio-8c6d693d4428e29679046526389593a499a8d157587c09dbc94e229d54117829 WatchSource:0}: Error finding container 8c6d693d4428e29679046526389593a499a8d157587c09dbc94e229d54117829: Status 404 returned error can't find the container with id 8c6d693d4428e29679046526389593a499a8d157587c09dbc94e229d54117829
Oct 30 11:43:36.440006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:36.439966    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" event={"ID":"c8b26e16-f0aa-4d69-ab7e-6692ad20c506","Type":"ContainerStarted","Data":"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"}
Oct 30 11:43:36.440006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:36.440011    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" event={"ID":"c8b26e16-f0aa-4d69-ab7e-6692ad20c506","Type":"ContainerStarted","Data":"8c6d693d4428e29679046526389593a499a8d157587c09dbc94e229d54117829"}
Oct 30 11:43:36.480068 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:36.480010    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podStartSLOduration=2.479992004 podStartE2EDuration="2.479992004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:43:36.477631462 +0000 UTC m=+815.697916369" watchObservedRunningTime="2026-10-30 11:43:36.479992004 +0000 UTC m=+815.700276915"
Oct 30 11:43:37.203118 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:37.203085    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:37.570037 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:43:37.569959    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:43:45.203489 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:45.203457    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:45.216062 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:45.216031    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:43:45.497092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:43:45.497014    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:44:07.127998 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.127963    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 11:44:07.133400 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.133379    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.156017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.155982    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"openshift-service-ca.crt\""
Oct 30 11:44:07.156017 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.156008    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"kube-root-ca.crt\""
Oct 30 11:44:07.156200 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.156145    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"default-dockercfg-qr7jz\""
Oct 30 11:44:07.156566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.156547    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e43a66\"/\"premium-w1-e43a66-kserve-self-signed-certs\""
Oct 30 11:44:07.162550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.162521    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 11:44:07.248265 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248231    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.248440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248272    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.248440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248382    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.248440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248417    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5rl\" (UniqueName: \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.248555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248507    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.248555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.248539    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349500    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349541    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349570    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349605    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5rl\" (UniqueName: \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349672    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349930 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349875    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.349989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.349977    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.350116 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.350085    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.350215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.350196    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.351938 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.351916    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.352107 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.352092    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.371485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.371455    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5rl\" (UniqueName: \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\") pod \"premium-w1-e43a66-kserve-5bdb95c45d-lnjnq\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") " pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.447992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.447956    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:07.571595 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:44:07.571557    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:44:07.778212 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:07.778181    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 11:44:07.779641 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:44:07.779612    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99db1f4a_8b87_4036_acca_bb160b1580d2.slice/crio-4771183838ac210fd6bb144f7e580d79f9bdce79e552cd142f980440f390412d WatchSource:0}: Error finding container 4771183838ac210fd6bb144f7e580d79f9bdce79e552cd142f980440f390412d: Status 404 returned error can't find the container with id 4771183838ac210fd6bb144f7e580d79f9bdce79e552cd142f980440f390412d
Oct 30 11:44:08.621992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:08.621950    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" event={"ID":"99db1f4a-8b87-4036-acca-bb160b1580d2","Type":"ContainerStarted","Data":"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"}
Oct 30 11:44:08.622410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:08.622004    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" event={"ID":"99db1f4a-8b87-4036-acca-bb160b1580d2","Type":"ContainerStarted","Data":"4771183838ac210fd6bb144f7e580d79f9bdce79e552cd142f980440f390412d"}
Oct 30 11:44:08.661231 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:08.661182    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podStartSLOduration=1.6611682110000001 podStartE2EDuration="1.661168211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:08.659653438 +0000 UTC m=+847.879938348" watchObservedRunningTime="2026-10-30 11:44:08.661168211 +0000 UTC m=+847.881453120"
Oct 30 11:44:09.448446 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:09.448406    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:17.448220 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:17.448119    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:17.465205 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:17.465172    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:17.686355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:17.685362    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:44:37.568836 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:44:37.568799    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:44:49.061687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.061651    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 11:44:49.065363 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.065319    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.078891 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.078865    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4e261b-openshift-default-dockercfg-slt8g\""
Oct 30 11:44:49.093512 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.093485    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 11:44:49.135865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.135819    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.135880    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136013    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136301 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136068    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136301 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136180    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh54\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136301 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136213    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136379    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136447    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.136509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.136478    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237383    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237463    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237497    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237577    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237609    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237658    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237693    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237722    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh54\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.237869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237747    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.238075 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237863    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.238075 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.237931    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.238212 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.238190    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.238291 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.238269    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.238509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.238485    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.240100 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.240075    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.240557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.240534    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.276509 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.276476    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh54\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.276873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.276851    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.379839 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.379810    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:49.694261 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.694234    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 11:44:49.696241 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:44:49.696212    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f3860b_0568_42a4_aba9_bc0876ef24b5.slice/crio-1a9f996b1f953029c25903fed848df0b6e93fe14e8efaa9de84f378243cadc67 WatchSource:0}: Error finding container 1a9f996b1f953029c25903fed848df0b6e93fe14e8efaa9de84f378243cadc67: Status 404 returned error can't find the container with id 1a9f996b1f953029c25903fed848df0b6e93fe14e8efaa9de84f378243cadc67
Oct 30 11:44:49.698599 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.698566    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:44:49.698660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.698634    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:44:49.698703 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.698665    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:44:49.839058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.839024    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" event={"ID":"24f3860b-0568-42a4-aba9-bc0876ef24b5","Type":"ContainerStarted","Data":"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"}
Oct 30 11:44:49.839238 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.839076    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" event={"ID":"24f3860b-0568-42a4-aba9-bc0876ef24b5","Type":"ContainerStarted","Data":"1a9f996b1f953029c25903fed848df0b6e93fe14e8efaa9de84f378243cadc67"}
Oct 30 11:44:49.880746 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:49.880691    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podStartSLOduration=0.880677645 podStartE2EDuration="880.677645ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:44:49.879306444 +0000 UTC m=+889.099591356" watchObservedRunningTime="2026-10-30 11:44:49.880677645 +0000 UTC m=+889.100962554"
Oct 30 11:44:50.380641 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:50.380608    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:50.382119 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:50.382095    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:50.382242 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:50.382148    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:51.380761 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:51.380720    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:51.381164 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:51.380787    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:52.381107 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:52.381076    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:52.381514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:52.381126    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:53.380591 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:53.380551    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:53.380790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:53.380608    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:54.380281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:54.380248    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:54.380702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:54.380304    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:55.380203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:55.380164    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 11:44:55.380406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:55.380217    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 11:44:56.384393 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:56.384359    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:56.384869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:56.384656    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:44:56.385424 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:44:56.385405    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 11:45:07.570054 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:45:07.570017    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:45:37.658612 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:45:37.658572    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:46:04.601588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.601282    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 11:46:04.605791 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.605766    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.630504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.630475    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"default-dockercfg-sxhks\""
Oct 30 11:46:04.630679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.630661    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"openshift-service-ca.crt\""
Oct 30 11:46:04.630763 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.630663    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"kube-root-ca.crt\""
Oct 30 11:46:04.630818 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.630756    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-1d9b1a\"/\"distinct-2-w2-1d9b1a-kserve-self-signed-certs\""
Oct 30 11:46:04.633823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.633791    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 11:46:04.739352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739299    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.739538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739400    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljvr\" (UniqueName: \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.739538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739485    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.739538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739521    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.739668 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739591    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.739668 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.739639    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840183 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840151    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840183 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840188    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840239    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840267    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljvr\" (UniqueName: \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840323    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840361    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840602    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840638    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.840775 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.840704    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.842530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.842510    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.842784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.842767    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.861344 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.861311    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljvr\" (UniqueName: \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\") pod \"distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") " pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:04.918326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:04.918292    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:05.220125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:05.220089    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 11:46:05.221158 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:46:05.221133    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0208e127_6f17_4552_bd62_26cdaed45cf4.slice/crio-a69e7ca0abbd65c340face52a5b79d48697dd72d83e6b59cbfe0c2929904380a WatchSource:0}: Error finding container a69e7ca0abbd65c340face52a5b79d48697dd72d83e6b59cbfe0c2929904380a: Status 404 returned error can't find the container with id a69e7ca0abbd65c340face52a5b79d48697dd72d83e6b59cbfe0c2929904380a
Oct 30 11:46:06.226324 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:06.226285    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" event={"ID":"0208e127-6f17-4552-bd62-26cdaed45cf4","Type":"ContainerStarted","Data":"1e3db5f38da1514a66432669b5b5eff37d9f28b19a984f5786c1c5ae4ddc21e7"}
Oct 30 11:46:06.226736 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:06.226345    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" event={"ID":"0208e127-6f17-4552-bd62-26cdaed45cf4","Type":"ContainerStarted","Data":"a69e7ca0abbd65c340face52a5b79d48697dd72d83e6b59cbfe0c2929904380a"}
Oct 30 11:46:06.264374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:06.264255    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podStartSLOduration=2.264240885 podStartE2EDuration="2.264240885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:06.262239954 +0000 UTC m=+965.482524864" watchObservedRunningTime="2026-10-30 11:46:06.264240885 +0000 UTC m=+965.484525795"
Oct 30 11:46:06.918598 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:06.918567    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:07.569830 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:46:07.569794    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:46:14.407161 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.407131    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 11:46:14.411550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.411526    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.433080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.433056    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4e261b\"/\"distinct-2-w3-4e261b-kserve-self-signed-certs\""
Oct 30 11:46:14.436156 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.436132    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 11:46:14.556051 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556014    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.556215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556087    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.556215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556119    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrkd\" (UniqueName: \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.556322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556241    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.556897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556420    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.556897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.556520    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657503    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657563    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657583    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrkd\" (UniqueName: \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657616    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657681    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.657749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.657720    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.658112 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.658075    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.658260 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.658117    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.658260 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.658087    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.659776 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.659746    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.660138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.660120    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.679767 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.679735    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrkd\" (UniqueName: \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\") pod \"distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") " pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.724792 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.724759    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:14.919371 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.919279    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:14.931902 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:14.931879    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:15.020799 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.020771    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 11:46:15.023035 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:46:15.023011    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1641683c_5de9_4297_87d5_75276e757516.slice/crio-915abd029970e80fbeb53251988ed1675175030be0691178a3857e85afab078b WatchSource:0}: Error finding container 915abd029970e80fbeb53251988ed1675175030be0691178a3857e85afab078b: Status 404 returned error can't find the container with id 915abd029970e80fbeb53251988ed1675175030be0691178a3857e85afab078b
Oct 30 11:46:15.319714 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.319629    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" event={"ID":"1641683c-5de9-4297-87d5-75276e757516","Type":"ContainerStarted","Data":"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"}
Oct 30 11:46:15.319714 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.319669    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" event={"ID":"1641683c-5de9-4297-87d5-75276e757516","Type":"ContainerStarted","Data":"915abd029970e80fbeb53251988ed1675175030be0691178a3857e85afab078b"}
Oct 30 11:46:15.331243 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.331220    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:46:15.356798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.356750    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podStartSLOduration=1.356737929 podStartE2EDuration="1.356737929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:15.355766693 +0000 UTC m=+974.576051602" watchObservedRunningTime="2026-10-30 11:46:15.356737929 +0000 UTC m=+974.577022839"
Oct 30 11:46:15.725881 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:15.725841    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:24.725038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:24.724996    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:24.738459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:24.738432    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:25.383714 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:25.383683    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:46:30.234092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.234061    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 11:46:30.237790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.237765    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.250559 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.250536    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-849669-openshift-default-dockercfg-2qnx2\""
Oct 30 11:46:30.266733 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.266709    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 11:46:30.322770 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.322739    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.322912 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.322794    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.322912 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.322898    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323005 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.322927    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323005 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.322951    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323134 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.323117    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.323151    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323192 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.323184    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78s57\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.323270 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.323246    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424033 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424006    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424043    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424079    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424180    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424215    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424273    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424294    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424480 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424313    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78s57\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424480 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424376    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424476    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424772 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424720    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424772 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424740    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424837    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.424970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.424859    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.426505 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.426485    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.426757 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.426736    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.444862 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.444840    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.445086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.445068    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78s57\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.552185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.552119    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:30.868490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.867930    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 11:46:30.868685 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:46:30.868551    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc750f8f8_84c6_4e9f_aaf7_b867db5d5c90.slice/crio-6937e612735bf22ef2cf343362d1bb46304b281b9878179fc2aabd059be1ded0 WatchSource:0}: Error finding container 6937e612735bf22ef2cf343362d1bb46304b281b9878179fc2aabd059be1ded0: Status 404 returned error can't find the container with id 6937e612735bf22ef2cf343362d1bb46304b281b9878179fc2aabd059be1ded0
Oct 30 11:46:30.871268 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.871223    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:30.871432 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.871316    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:30.871432 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:30.871375    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:31.420931 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.420884    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" event={"ID":"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90","Type":"ContainerStarted","Data":"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"}
Oct 30 11:46:31.421344 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.420947    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" event={"ID":"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90","Type":"ContainerStarted","Data":"6937e612735bf22ef2cf343362d1bb46304b281b9878179fc2aabd059be1ded0"}
Oct 30 11:46:31.493856 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.493783    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podStartSLOduration=1.4937628250000001 podStartE2EDuration="1.493762825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:31.482363918 +0000 UTC m=+990.702648828" watchObservedRunningTime="2026-10-30 11:46:31.493762825 +0000 UTC m=+990.714047736"
Oct 30 11:46:31.553093 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.553059    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:31.554892 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.554867    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:31.555029 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:31.554930    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:32.552660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:32.552621    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:32.553058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:32.552677    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:33.552455 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:33.552420    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:33.552630 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:33.552470    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:34.353611 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.353577    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 11:46:34.357764 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.357745    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.408895 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.389815    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 11:46:34.472035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472001    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbp8\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472044    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472129    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472288 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472244    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472389 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472372    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472445 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472409    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472445 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472433    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472513 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472495    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.472552 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.472522    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.553374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.553321    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:34.553540 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.553397    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:34.573794 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573768    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbp8\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.573917 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573806    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.573917 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573852    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.573917 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573905    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573967    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.573998    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574021    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574061    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574095    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574204    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574230    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574429 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574407    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574607 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574587    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.574863 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.574843    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.576308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.576288    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.576597 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.576581    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.616039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.616002    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.617756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.617733    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbp8\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:34.672910 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:34.672888    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:35.017684 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.017659    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 11:46:35.019890 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:46:35.019864    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9483385_92b9_4b99_8f17_9b554199e2cf.slice/crio-d98c45db7f9e1e70a599dd2277f50e0dfe5ac2af385dff7d70e62820ef8de1dd WatchSource:0}: Error finding container d98c45db7f9e1e70a599dd2277f50e0dfe5ac2af385dff7d70e62820ef8de1dd: Status 404 returned error can't find the container with id d98c45db7f9e1e70a599dd2277f50e0dfe5ac2af385dff7d70e62820ef8de1dd
Oct 30 11:46:35.022207 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.022179    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:35.022262 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.022238    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:35.022298 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.022271    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:35.443839 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.443798    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" event={"ID":"c9483385-92b9-4b99-8f17-9b554199e2cf","Type":"ContainerStarted","Data":"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"}
Oct 30 11:46:35.444341 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.443871    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" event={"ID":"c9483385-92b9-4b99-8f17-9b554199e2cf","Type":"ContainerStarted","Data":"d98c45db7f9e1e70a599dd2277f50e0dfe5ac2af385dff7d70e62820ef8de1dd"}
Oct 30 11:46:35.510352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.510054    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podStartSLOduration=1.510033604 podStartE2EDuration="1.510033604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:35.494767317 +0000 UTC m=+994.715052228" watchObservedRunningTime="2026-10-30 11:46:35.510033604 +0000 UTC m=+994.730318516"
Oct 30 11:46:35.554359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.553931    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:35.554359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.554007    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:35.673465 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:35.673433    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:36.553540 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:36.553504    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:36.553969 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:36.553561    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:36.674279 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:36.674249    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:36.674519 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:36.674496    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:37.553473 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:37.553433    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 11:46:37.553624 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:37.553499    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 11:46:37.575165 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:46:37.575106    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:46:37.673485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:37.673445    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:37.673657 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:37.673522    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:38.557158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:38.557128    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:38.557554 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:38.557429    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:38.558061 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:38.558039    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 11:46:38.673313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:38.673284    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:38.673491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:38.673357    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:39.674181 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:39.674145    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:39.674676 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:39.674212    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:40.673897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:40.673848    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:40.674080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:40.673936    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:41.674261 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:41.674220    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 11:46:41.674739 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:41.674279    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 11:46:42.677956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:42.677927    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:42.678303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:42.678288    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:42.679067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:42.679044    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 11:46:54.083984 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.083948    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 11:46:54.087913 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.087891    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.104899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.104876    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e43a66-openshift-default-dockercfg-pvppc\""
Oct 30 11:46:54.112717 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.112690    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 11:46:54.134018 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.133993    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134162 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134032    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134162 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134062    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qnb2\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134287 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134205    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134287 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134251    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134434 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134311    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134498 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134440    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134556 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134531    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.134651 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.134624    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235632 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235600    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235644    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235698    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235735    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235779    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.235843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235816    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235856    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qnb2\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235903    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.235938    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.236211    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.236235    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236537 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.236280    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.236778 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.236755    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.237082 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.237057    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.238461 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.238439    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.238679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.238660    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.256719 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.256699    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.256885 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.256868    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qnb2\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.401404 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.401367    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:54.732729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.732685    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 11:46:54.733556 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:46:54.733522    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdc5f2a_7e14_473b_b459_8b78e39b0be8.slice/crio-08283dfa91edff31f6f7b9d707c3c90d00434c359ddc5a5a2c1f69a566a28076 WatchSource:0}: Error finding container 08283dfa91edff31f6f7b9d707c3c90d00434c359ddc5a5a2c1f69a566a28076: Status 404 returned error can't find the container with id 08283dfa91edff31f6f7b9d707c3c90d00434c359ddc5a5a2c1f69a566a28076
Oct 30 11:46:54.736110 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.736074    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:54.736214 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.736143    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:54.736214 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:54.736174    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:46:55.556409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:55.556356    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" event={"ID":"ebdc5f2a-7e14-473b-b459-8b78e39b0be8","Type":"ContainerStarted","Data":"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"}
Oct 30 11:46:55.556803 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:55.556432    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" event={"ID":"ebdc5f2a-7e14-473b-b459-8b78e39b0be8","Type":"ContainerStarted","Data":"08283dfa91edff31f6f7b9d707c3c90d00434c359ddc5a5a2c1f69a566a28076"}
Oct 30 11:46:55.615631 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:55.615567    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podStartSLOduration=1.615545725 podStartE2EDuration="1.615545725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:46:55.612640148 +0000 UTC m=+1014.832925070" watchObservedRunningTime="2026-10-30 11:46:55.615545725 +0000 UTC m=+1014.835830637"
Oct 30 11:46:56.414540 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:56.414504    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:46:56.416019 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:56.415992    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:46:56.416148 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:56.416048    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 11:46:57.411356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:57.411153    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:46:57.411356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:57.411213    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 11:46:58.406710 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:58.406672    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:46:58.407035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:58.407003    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 11:46:59.411804 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:59.411760    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:46:59.413250 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:46:59.413212    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 11:47:00.402739 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:00.402701    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 11:47:00.403012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:00.402981    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 11:47:01.751838 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:01.751801    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:47:01.752868 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:01.752831    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:47:01.754174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:01.754148    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 11:47:07.569631 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:47:07.569592    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:47:18.053154 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.053072    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 11:47:18.058586 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.058564    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.072967 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.072945    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1d9b1a-openshift-default-dockercfg-p2zdz\""
Oct 30 11:47:18.091675 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.091644    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 11:47:18.141510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141477    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141556    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141635    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141659    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141686    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141743    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnqh\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141847    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.141924 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141890    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.142027 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.141950    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243229 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243186    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243256    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243290    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243307    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243325    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243381    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnqh\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243442    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243486    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243517    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243928 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243833    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243928 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243879    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.243928 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243908    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.244099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.243955    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.244518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.244488    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.245749 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.245724    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.245968 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.245949    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.265979 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.265936    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnqh\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.266299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.266272    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.371858 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.371818    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:18.696140 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.696101    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 11:47:18.697683 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:47:18.697618    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe166ae9_5782_4efe_81e0_ac9d359e1c33.slice/crio-488cf60518866dc4b16bb77acca4fadca42f186912baba15ea38c5373e6a80b6 WatchSource:0}: Error finding container 488cf60518866dc4b16bb77acca4fadca42f186912baba15ea38c5373e6a80b6: Status 404 returned error can't find the container with id 488cf60518866dc4b16bb77acca4fadca42f186912baba15ea38c5373e6a80b6
Oct 30 11:47:18.700293 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.700255    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:18.700441 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.700396    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:18.700510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:18.700447    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:19.550355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.549201    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:47:19.574357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.569560    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.587196 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.582862    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700072    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700157    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700200    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700274    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700298    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700348    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mxl\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700392    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700438    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.702354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.700534    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.764352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.763439    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" event={"ID":"fe166ae9-5782-4efe-81e0-ac9d359e1c33","Type":"ContainerStarted","Data":"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"}
Oct 30 11:47:19.764352 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.763506    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" event={"ID":"fe166ae9-5782-4efe-81e0-ac9d359e1c33","Type":"ContainerStarted","Data":"488cf60518866dc4b16bb77acca4fadca42f186912baba15ea38c5373e6a80b6"}
Oct 30 11:47:19.803410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.803297    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.803600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.803447    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.803676 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.803607    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.804285    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.804882    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.805876    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.805479    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.805946    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.805980    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mxl\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.806979    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.807048    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807909 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.807444    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.807909 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.807700    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.855522 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.855103    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.859154 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.858414    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mxl\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.859464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.859438    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.865371 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.864781    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.865371 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.865074    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:19.957671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:19.957435    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:20.417361 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.389625    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:20.653350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.652845    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:20.653350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.652903    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:20.667703 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.667642    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podStartSLOduration=2.667621395 podStartE2EDuration="2.667621395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:19.85778173 +0000 UTC m=+1039.078066641" watchObservedRunningTime="2026-10-30 11:47:20.667621395 +0000 UTC m=+1039.887906307"
Oct 30 11:47:20.668664 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.668631    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:47:20.680056 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:47:20.680032    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3fc2b8_93f3_4461_b8f5_4d1ded82b3da.slice/crio-287a94859ee97ff5d0ca78f863e5ab22e2fdff46d2fae7bb698888244b27effc WatchSource:0}: Error finding container 287a94859ee97ff5d0ca78f863e5ab22e2fdff46d2fae7bb698888244b27effc: Status 404 returned error can't find the container with id 287a94859ee97ff5d0ca78f863e5ab22e2fdff46d2fae7bb698888244b27effc
Oct 30 11:47:20.683282 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.683248    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:20.683412 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.683346    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:20.683412 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.683397    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:20.781572 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:20.781480    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" event={"ID":"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da","Type":"ContainerStarted","Data":"287a94859ee97ff5d0ca78f863e5ab22e2fdff46d2fae7bb698888244b27effc"}
Oct 30 11:47:21.380378 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.372285    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:21.380378 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.372367    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:21.843813 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.843619    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" event={"ID":"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da","Type":"ContainerStarted","Data":"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"}
Oct 30 11:47:21.940966 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.940905    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podStartSLOduration=2.9408824989999998 podStartE2EDuration="2.940882499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:21.928698736 +0000 UTC m=+1041.148983683" watchObservedRunningTime="2026-10-30 11:47:21.940882499 +0000 UTC m=+1041.161167409"
Oct 30 11:47:21.958215 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.958186    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:21.959520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.959497    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:21.959692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:21.959674    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:22.373085 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:22.373045    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:22.373274 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:22.373118    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:22.958160 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:22.958128    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:22.958565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:22.958187    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:23.372413 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:23.372376    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:23.372609 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:23.372432    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:23.958156 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:23.958125    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:23.958725 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:23.958182    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:24.372828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:24.372801    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:24.373062 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:24.373042    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:24.958761 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:24.958723    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:24.959268 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:24.958799    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:25.372955 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:25.372924    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 11:47:25.373128 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:25.372984    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 11:47:25.958540 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:25.958496    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:25.958751 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:25.958551    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:26.437830 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:26.437797    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:26.438272 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:26.438177    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:26.439119 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:26.439097    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 11:47:26.958150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:26.958118    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:26.958313 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:26.958168    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:27.958178 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:27.958144    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 11:47:27.958602 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:27.958209    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 11:47:28.962535 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:28.962506    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:28.963025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:28.963008    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:28.963717 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:28.963691    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:47:37.571881 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:47:37.571844    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:47:44.167847 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.167812    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 11:47:44.173781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.173757    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.196436 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.196404    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 11:47:44.207312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.207283    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3ccdab\"/\"distinct-2-w0-3ccdab-kserve-self-signed-certs\""
Oct 30 11:47:44.248487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248460    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.248487 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248495    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.248673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248513    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.248673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248615    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v222b\" (UniqueName: \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.248742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248701    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.248779 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.248761    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350302    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v222b\" (UniqueName: \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350393    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350483    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350580 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350560    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350617 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350581    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350617 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350602    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.350937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350914    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.351007 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.350974    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.351049 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.351008    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.353108 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.353082    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.353233 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.353082    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.382027 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.381996    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v222b\" (UniqueName: \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\") pod \"distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") " pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.502925 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.501604    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:44.806621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:44.805436    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 11:47:44.808753 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:47:44.808719    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7f5ef93_6f28_4ed2_a7a4_148edbb9f9fc.slice/crio-6667484e0a8d4a8b400f5c497dc4276e588aff3780c7d5c0b3856bfb71a633ea WatchSource:0}: Error finding container 6667484e0a8d4a8b400f5c497dc4276e588aff3780c7d5c0b3856bfb71a633ea: Status 404 returned error can't find the container with id 6667484e0a8d4a8b400f5c497dc4276e588aff3780c7d5c0b3856bfb71a633ea
Oct 30 11:47:45.018817 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:45.018779    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" event={"ID":"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc","Type":"ContainerStarted","Data":"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"}
Oct 30 11:47:45.018973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:45.018832    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" event={"ID":"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc","Type":"ContainerStarted","Data":"6667484e0a8d4a8b400f5c497dc4276e588aff3780c7d5c0b3856bfb71a633ea"}
Oct 30 11:47:45.056455 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:45.056376    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podStartSLOduration=1.056364552 podStartE2EDuration="1.056364552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:45.055468811 +0000 UTC m=+1064.275753720" watchObservedRunningTime="2026-10-30 11:47:45.056364552 +0000 UTC m=+1064.276649462"
Oct 30 11:47:45.503693 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:45.503663    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:54.320304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.320261    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 11:47:54.324842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.324819    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.349144 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.349113    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 11:47:54.473889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.473848    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.473923    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.473978    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474057    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474193 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474104    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwl7\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474248 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474233    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474284 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474274    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474372 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474360    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.474418 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.474388    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.504045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.504009    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:54.576409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576196    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576277    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576357    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576392    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576506    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576578    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576610    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576640    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576699    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwl7\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.576756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.576740    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.577859 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.577823    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.582357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.579844    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.582357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.580065    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.582357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.580613    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.582842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.582817    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.583601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.583555    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.584294 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.584271    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:54.602038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.601981    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.606007 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.605968    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwl7\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:54.646527 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:54.646483    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:55.094394 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.090131    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 11:47:55.098417 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:47:55.096935    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e2f508_5903_41ac_babf_991306dcf37d.slice/crio-6cdefbbb279cc02c804e30d3eb306640f4070d397080f4887e931e7c7e0e8836 WatchSource:0}: Error finding container 6cdefbbb279cc02c804e30d3eb306640f4070d397080f4887e931e7c7e0e8836: Status 404 returned error can't find the container with id 6cdefbbb279cc02c804e30d3eb306640f4070d397080f4887e931e7c7e0e8836
Oct 30 11:47:55.099965 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.099897    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:55.100076 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.099991    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:55.100076 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.100036    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:47:55.111361 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.107547    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:47:55.887500 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.887468    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:47:55.888269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:55.888236    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy" containerID="cri-o://27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e" gracePeriod=30
Oct 30 11:47:56.221318 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.221212    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" event={"ID":"87e2f508-5903-41ac-babf-991306dcf37d","Type":"ContainerStarted","Data":"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"}
Oct 30 11:47:56.221617 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.221589    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" event={"ID":"87e2f508-5903-41ac-babf-991306dcf37d","Type":"ContainerStarted","Data":"6cdefbbb279cc02c804e30d3eb306640f4070d397080f4887e931e7c7e0e8836"}
Oct 30 11:47:56.281390 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.280149    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podStartSLOduration=2.2801298 podStartE2EDuration="2.2801298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:47:56.278822031 +0000 UTC m=+1075.499106942" watchObservedRunningTime="2026-10-30 11:47:56.2801298 +0000 UTC m=+1075.500414713"
Oct 30 11:47:56.477413 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.465624    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:56.477413 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.470404    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 11:47:56.522353 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.514402    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:47:56.650351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.650284    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:47:56.650550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.650468    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:47:56.650550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.650516    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:47:56.705383 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:56.704431    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.110356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.109769    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.212381 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.211816    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" podUID="7843c47a-0b1d-4004-a852-923b462f0c5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.364410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.363549    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.467356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.466399    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.467356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.467087    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:47:57.488355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.486823    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 11:47:57.488355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.487213    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:47:57.544594 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.519425    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.544594 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.534814    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:47:57.544989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.544956    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:47:57.545197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.545155    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.684871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.640477    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:47:57.684871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.682081    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 11:47:57.711289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.709373    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:47:57.711289 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.709432    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:47:57.739129 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.735590    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:47:57.778737 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:57.778710    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:47:58.647272 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:58.647230    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:47:58.647721 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:58.647298    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:47:59.647600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:59.647561    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:47:59.648002 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:47:59.647623    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:48:00.647545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:00.647506    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:48:00.647711 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:00.647572    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:48:01.176255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.176185    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:48:01.276326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.276285    4453 generic.go:356] "Generic (PLEG): container finished" podID="8718780e-49dd-4e5c-be31-527d4d6590df" containerID="27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e" exitCode=0
Oct 30 11:48:01.276545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.276377    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"
Oct 30 11:48:01.276545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.276392    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" event={"ID":"8718780e-49dd-4e5c-be31-527d4d6590df","Type":"ContainerDied","Data":"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"}
Oct 30 11:48:01.276545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.276447    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s" event={"ID":"8718780e-49dd-4e5c-be31-527d4d6590df","Type":"ContainerDied","Data":"67d5a4f6eb0cac72a5932b94fc5f4ff813a042d65e57ba7e0ccde44f40b78834"}
Oct 30 11:48:01.276545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.276474    4453 scope.go:122] "RemoveContainer" containerID="27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"
Oct 30 11:48:01.287396 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.287377    4453 scope.go:122] "RemoveContainer" containerID="27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"
Oct 30 11:48:01.287708 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:48:01.287682    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e\": container with ID starting with 27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e not found: ID does not exist" containerID="27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"
Oct 30 11:48:01.287845 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.287717    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e"} err="failed to get container status \"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e\": rpc error: code = NotFound desc = could not find container \"27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e\": container with ID starting with 27c289996f665ee2d1041c86d5510a3c9bf5e4240b8e0e53efb0bfddf9d3275e not found: ID does not exist"
Oct 30 11:48:01.331376 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331340    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331394    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331476    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331537    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331572    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331609    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331715 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331677    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331872 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331798    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331872 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331833    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\") pod \"8718780e-49dd-4e5c-be31-527d4d6590df\" (UID: \"8718780e-49dd-4e5c-be31-527d4d6590df\") "
Oct 30 11:48:01.331986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.331891    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:48:01.332476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332279    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:01.332599 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332494    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:01.332735 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332709    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8718780e-49dd-4e5c-be31-527d4d6590df-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.332735 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332738    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.332929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332794    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:01.332929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.332840    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:01.334454 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.334425    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:48:01.334970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.334939    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "kube-api-access-szvdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:01.335322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.335302    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:48:01.335410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.335370    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo" pod "8718780e-49dd-4e5c-be31-527d4d6590df" (UID: "8718780e-49dd-4e5c-be31-527d4d6590df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434136    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434166    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434175    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434187    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8718780e-49dd-4e5c-be31-527d4d6590df-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434199    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434206    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8718780e-49dd-4e5c-be31-527d4d6590df-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.434216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.434214    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvdw\" (UniqueName: \"kubernetes.io/projected/8718780e-49dd-4e5c-be31-527d4d6590df-kube-api-access-szvdw\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:48:01.625524 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.625495    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:48:01.635037 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.635009    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-rkv2s"]
Oct 30 11:48:01.647072 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.647042    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:48:01.647200 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:01.647091    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:48:02.647712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:02.647676    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 11:48:02.648092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:02.647735    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 11:48:02.912981 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:02.912909    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" path="/var/lib/kubelet/pods/8718780e-49dd-4e5c-be31-527d4d6590df/volumes"
Oct 30 11:48:03.202486 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.202406    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:48:03.202982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.202966    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy"
Oct 30 11:48:03.203024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.202984    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy"
Oct 30 11:48:03.203104 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.203094    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8718780e-49dd-4e5c-be31-527d4d6590df" containerName="istio-proxy"
Oct 30 11:48:03.207857 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.207835    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.235692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.235658    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:48:03.353518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353482    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353518 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353518    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353544    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hk8\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353693    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353798    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353874    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353913    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353932    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.353992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.353958    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.454790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454689    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.454790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454735    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.454790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454762    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454833    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454849    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454871    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hk8\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.454918    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455094 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455012    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455363 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455132    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455363 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455202    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455363 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455275    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455534 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455405    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455534 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455480    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.455958 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.455927    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.457576 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.457547    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.457704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.457685    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.476951 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.476918    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hk8\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.477086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.477069    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.523508 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.523476    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:03.857760 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:48:03.857730    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492ee220_3c69_4b52_ab7d_ec1371ee1c8c.slice/crio-b8cb73e58159e83cfada04b58dbc4224b2a9c00602e08b40cc97cf7629147739 WatchSource:0}: Error finding container b8cb73e58159e83cfada04b58dbc4224b2a9c00602e08b40cc97cf7629147739: Status 404 returned error can't find the container with id b8cb73e58159e83cfada04b58dbc4224b2a9c00602e08b40cc97cf7629147739
Oct 30 11:48:03.858162 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.857863    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:48:03.860584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.860540    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:03.860700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.860640    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:03.860700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.860690    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:03.878815 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.878776    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:48:03.879166 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.879142    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:48:03.882679 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:03.882658    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 11:48:04.304102 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.304064    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" event={"ID":"492ee220-3c69-4b52-ab7d-ec1371ee1c8c","Type":"ContainerStarted","Data":"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"}
Oct 30 11:48:04.304294 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.304148    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" event={"ID":"492ee220-3c69-4b52-ab7d-ec1371ee1c8c","Type":"ContainerStarted","Data":"b8cb73e58159e83cfada04b58dbc4224b2a9c00602e08b40cc97cf7629147739"}
Oct 30 11:48:04.344908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.344858    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podStartSLOduration=1.344846349 podStartE2EDuration="1.344846349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:04.343493497 +0000 UTC m=+1083.563778406" watchObservedRunningTime="2026-10-30 11:48:04.344846349 +0000 UTC m=+1083.565131259"
Oct 30 11:48:04.524277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.524241    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:04.525697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.525673    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:04.525830 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:04.525721    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:05.524154 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:05.524117    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:05.524578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:05.524181    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:06.526485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:06.526059    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:06.526485 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:06.526125    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:07.545367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:07.536749    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:07.545367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:07.536817    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:07.778379 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:48:07.774762    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:48:08.371354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.370456    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:08.478355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.467466    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:08.526355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.525715    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:08.526355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.525782    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:08.656410 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.655444    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:08.683172 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.683130    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:08.730522 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.730469    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:08.785640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:08.784441    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:48:09.092416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.091549    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 11:48:09.382907 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.382875    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:48:09.387506 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.387483    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.420089 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.420043    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:48:09.446085 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446047    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446254 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446213    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446319 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446267    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446408 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446389    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446455 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446439    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bxb\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446494 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446477    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446550 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446535    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446625 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446609    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.446663 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.446633    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.524923 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.524887    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:09.525150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.524945    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:09.547267 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547228    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547415 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547367    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547412    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547436    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547454    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bxb\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547491    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547510    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547557    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547572    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547711    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547731    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.547999 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.547977    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.548065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.548024    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.548287 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.548264    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.549781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.549759    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.550054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.550029    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.568190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.568163    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.568302 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.568288    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bxb\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\") pod \"e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") " pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:09.732917 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:09.732831    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:10.073164 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.073131    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:48:10.074882 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:48:10.074854    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491fc23e_5170_4eba_af16_371e67138787.slice/crio-702dc883fcaeb327df371eecb13d820069b5845b6832f5abb07cc7b21a32cf96 WatchSource:0}: Error finding container 702dc883fcaeb327df371eecb13d820069b5845b6832f5abb07cc7b21a32cf96: Status 404 returned error can't find the container with id 702dc883fcaeb327df371eecb13d820069b5845b6832f5abb07cc7b21a32cf96
Oct 30 11:48:10.077629 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.077593    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:10.077729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.077667    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:10.077729 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.077696    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:10.383427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.383378    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" event={"ID":"491fc23e-5170-4eba-af16-371e67138787","Type":"ContainerStarted","Data":"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"}
Oct 30 11:48:10.383629 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.383448    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" event={"ID":"491fc23e-5170-4eba-af16-371e67138787","Type":"ContainerStarted","Data":"702dc883fcaeb327df371eecb13d820069b5845b6832f5abb07cc7b21a32cf96"}
Oct 30 11:48:10.444920 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.444853    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podStartSLOduration=1.4448333500000001 podStartE2EDuration="1.44483335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:10.433414645 +0000 UTC m=+1089.653699568" watchObservedRunningTime="2026-10-30 11:48:10.44483335 +0000 UTC m=+1089.665118261"
Oct 30 11:48:10.524686 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.524651    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 11:48:10.524888 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.524702    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 11:48:10.733629 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:10.733560    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:11.735819 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:11.735786    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:11.736348 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:11.735849    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:12.565553 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.565513    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:48:12.565756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.565582    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:12.733902 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.733863    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:12.734080 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.733929    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:12.855916 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.855834    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:12.856450 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.856222    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:12.857527 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:12.857506    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:48:13.734092 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:13.734052    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:13.734292 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:13.734114    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:14.733606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:14.733569    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:14.734091 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:14.733638    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:15.733843 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:15.733807    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:15.734241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:15.733866    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:16.733778 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:16.733746    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 11:48:16.733962 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:16.733816    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 11:48:17.738528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:17.738497    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:17.739058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:17.738934    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:17.741182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:17.741160    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:48:26.207755 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.205521    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" podUID="7843c47a-0b1d-4004-a852-923b462f0c5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:26.275355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.274449    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:26.327355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.326427    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:26.658356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.656467    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 11:48:26.773355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.767407    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:48:26.773355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.767780    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:26.818355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:26.807135    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 30 11:48:27.132417 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.081581    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.284355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.258139    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" podUID="7843c47a-0b1d-4004-a852-923b462f0c5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded"
Oct 30 11:48:27.284355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.275177    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:48:27.350355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.341424    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.350355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.342220    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:48:27.450836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.450741    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:48:27.470355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.451074    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.495484 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.495436    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 11:48:27.495681 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.495523    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.656840 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.656799    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.670751 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.661636    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.712295 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.671494    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:48:27.728907 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.728878    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 11:48:27.845054 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.844995    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:27.848045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.848023    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:48:27.848997 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.845664    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 11:48:27.849986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:27.849956    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:48:28.182417 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.180180    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:48:28.200360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.183229    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:48:28.210474 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.210446    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 11:48:28.211948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.211925    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 11:48:28.268634 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.267806    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f"
Oct 30 11:48:28.268634 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.267880    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 11:48:28.276288 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:28.276266    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 11:48:33.328587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.326505    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:48:33.336110 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.336077    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.361582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.361553    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:48:33.445915 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.445872    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446120 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.445974    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnw8\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446120 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446012    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446120 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446039    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446120 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446108    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446366 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446131    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446366 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446278    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446449    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.446476 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.446471    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547519    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547568    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547639    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547665    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnw8\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547834 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547794    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547837    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547900    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.547970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.547924    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548133 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548052    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548133 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548098    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548257    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548364 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548282    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548430 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548389    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.548562 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.548539    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.550167 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.550147    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.550271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.550214    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.568655 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.568633    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.568999 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.568978    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnw8\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.656733 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.656708    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:33.978691 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.978660    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:48:33.979228 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:48:33.979203    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fcdbf9_9d8d_4be0_8e82_172e6d266ef4.slice/crio-76a63d424a6683bbff43f37d818a09a939b36c7d39787376cd11b2bf19b81911 WatchSource:0}: Error finding container 76a63d424a6683bbff43f37d818a09a939b36c7d39787376cd11b2bf19b81911: Status 404 returned error can't find the container with id 76a63d424a6683bbff43f37d818a09a939b36c7d39787376cd11b2bf19b81911
Oct 30 11:48:33.981587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.981553    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:33.981698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.981621    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:33.981698 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:33.981651    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:34.283484 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:34.283384    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" event={"ID":"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4","Type":"ContainerStarted","Data":"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"}
Oct 30 11:48:34.283787 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:34.283762    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" event={"ID":"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4","Type":"ContainerStarted","Data":"76a63d424a6683bbff43f37d818a09a939b36c7d39787376cd11b2bf19b81911"}
Oct 30 11:48:34.328890 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:34.328836    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podStartSLOduration=1.328820668 podStartE2EDuration="1.328820668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:34.323192903 +0000 UTC m=+1113.543477812" watchObservedRunningTime="2026-10-30 11:48:34.328820668 +0000 UTC m=+1113.549105578"
Oct 30 11:48:34.656958 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:34.656920    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:35.658702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:35.658665    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:35.659086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:35.658731    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:36.657397 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:36.657359    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:36.657599 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:36.657414    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:37.569854 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:48:37.569812    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:48:37.657691 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:37.657653    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:37.657853 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:37.657716    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:38.657437 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:38.657401    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:38.657930 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:38.657467    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:39.657690 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:39.657655    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:39.658162 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:39.657710    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:40.657021 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:40.656985    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 11:48:40.657197 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:40.657052    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 11:48:41.661415 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:41.661387    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:41.661840 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:41.661780    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:41.662481 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:41.662458    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:48:56.137052 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.136979    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 11:48:56.141440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.141417    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.183535 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.183494    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 11:48:56.305808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.305767    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.305808 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.305809    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzdp\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306083 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.305902    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306083 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.305965    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306083 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.306048    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306221 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.306155    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306283 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.306266    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306325 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.306290    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.306325 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.306308    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407262 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407185    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407262 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407227    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407262 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407248    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407311    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407368    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzdp\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407411    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407480    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407516    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407894 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407595    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407894 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407662    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.407894 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407888    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.408045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407965    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.408045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.407993    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.408184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.408163    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.409726 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.409702    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.409948 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.409929    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.428300 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.428271    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzdp\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.435255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.435229    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6tssr\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.455530 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.455508    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:56.772557 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.772528    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 11:48:56.773088 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:48:56.773063    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f99652_ea34_4cb3_ad33_55471d784e9b.slice/crio-eb1ebb8b536803c8f3cfe7c0515223f7835047a5a399e0ad0735ff9651cbb4fd WatchSource:0}: Error finding container eb1ebb8b536803c8f3cfe7c0515223f7835047a5a399e0ad0735ff9651cbb4fd: Status 404 returned error can't find the container with id eb1ebb8b536803c8f3cfe7c0515223f7835047a5a399e0ad0735ff9651cbb4fd
Oct 30 11:48:56.775555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.775520    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:56.775646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.775588    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:56.775646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:56.775620    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:48:57.529666 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:57.529627    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" event={"ID":"27f99652-ea34-4cb3-ad33-55471d784e9b","Type":"ContainerStarted","Data":"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"}
Oct 30 11:48:57.530086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:57.529680    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" event={"ID":"27f99652-ea34-4cb3-ad33-55471d784e9b","Type":"ContainerStarted","Data":"eb1ebb8b536803c8f3cfe7c0515223f7835047a5a399e0ad0735ff9651cbb4fd"}
Oct 30 11:48:57.616359 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:57.615522    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podStartSLOduration=1.615498461 podStartE2EDuration="1.615498461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:48:57.579586801 +0000 UTC m=+1136.799871733" watchObservedRunningTime="2026-10-30 11:48:57.615498461 +0000 UTC m=+1136.835783372"
Oct 30 11:48:58.455640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:58.455602    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:48:58.457506 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:58.457479    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:48:58.457633 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:58.457531    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:48:59.456298 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:59.456262    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:48:59.456704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:48:59.456322    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:49:00.456635 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:00.456600    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:49:00.457023 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:00.456655    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:49:01.457547 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:01.457504    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:49:01.458084 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:01.457574    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:49:01.972358 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:01.968142    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:49:01.983070 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:01.983037    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:01.989602 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:01.989569    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:49:02.002811 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.002768    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-71af6f-openshift-default-dockercfg-bhx98\""
Oct 30 11:49:02.078880 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.078838    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079139 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079119    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079273 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079255    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079464 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079439    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079477    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079661 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079623    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079724 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079699    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079781 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079746    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.079886 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.079864    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.180986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.180952    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.180986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.180989    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181035    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181073    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181099    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181167    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181221    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181236    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181251    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181684 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181646    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.181789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181720    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.182004 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.181988    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.182125 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.182102    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.182169 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.182151    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.183739 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.183723    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.183962 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.183945    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.230428 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.229963    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.231501 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.231473    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\") pod \"e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") " pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.326241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.326196    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:02.456784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.456606    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:49:02.456784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.456652    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:49:02.685006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.684973    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:49:02.685969 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:02.685941    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1eaf45_4d7c_4ceb_8019_0707cfcfb6b1.slice/crio-8497877b4cebd794d3372f657237b1e7c5a99569d955ca4b14b448d5fb410ac8 WatchSource:0}: Error finding container 8497877b4cebd794d3372f657237b1e7c5a99569d955ca4b14b448d5fb410ac8: Status 404 returned error can't find the container with id 8497877b4cebd794d3372f657237b1e7c5a99569d955ca4b14b448d5fb410ac8
Oct 30 11:49:02.691952 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.688598    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:02.691952 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.688684    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:02.691952 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:02.688727    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:03.456850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:03.456816    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6tssr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 11:49:03.457030 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:03.456876    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 11:49:03.563706 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:03.563666    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerStarted","Data":"79ebf17442c084bd23a536bb52c8473c6e5f872315d84edf2201db812232e84c"}
Oct 30 11:49:03.563893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:03.563733    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerStarted","Data":"8497877b4cebd794d3372f657237b1e7c5a99569d955ca4b14b448d5fb410ac8"}
Oct 30 11:49:03.614822 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:03.614759    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podStartSLOduration=2.61474586 podStartE2EDuration="2.61474586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:03.613086294 +0000 UTC m=+1142.833371205" watchObservedRunningTime="2026-10-30 11:49:03.61474586 +0000 UTC m=+1142.835030769"
Oct 30 11:49:04.326801 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.326765    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:04.332208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.332184    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:04.419577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.419541    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:49:04.424084 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.424062    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.437742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.437710    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d9258eb\"/\"e2e-lifecycle-model-6d9258eb-kserve-self-signed-certs\""
Oct 30 11:49:04.438458 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.438438    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6d9258eb\"/\"default-dockercfg-gggfn\""
Oct 30 11:49:04.440372 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.438890    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d9258eb\"/\"openshift-service-ca.crt\""
Oct 30 11:49:04.440372 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.438971    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6d9258eb\"/\"kube-root-ca.crt\""
Oct 30 11:49:04.451989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.451960    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:49:04.521721 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.521689    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.521902 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.521754    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.521946 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.521885    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.521996 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.521979    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.522124 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.522099    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntwgb\" (UniqueName: \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.522208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.522192    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.570765 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.570723    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:04.571833 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.571812    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:04.623500 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623467    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.623700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623573    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.623700 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623617    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.623824 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623710    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.623824 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623796    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntwgb\" (UniqueName: \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.623937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.623890    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.624137 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.624111    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.625020 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.624996    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.625184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.625035    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.656584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.628013    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.656584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.629931    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.656584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.649095    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntwgb\" (UniqueName: \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\") pod \"e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") " pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:04.755731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:04.755693    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:05.075486 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.075397    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:49:05.079544 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:05.079501    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f8c531_d78b_45ac_a252_dec3ba7d0cc5.slice/crio-6c46726c947207a6a80c37fe99486b74b625d14ea8d00d3b66d62d7bb913f844 WatchSource:0}: Error finding container 6c46726c947207a6a80c37fe99486b74b625d14ea8d00d3b66d62d7bb913f844: Status 404 returned error can't find the container with id 6c46726c947207a6a80c37fe99486b74b625d14ea8d00d3b66d62d7bb913f844
Oct 30 11:49:05.324525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.324482    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:49:05.324905 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.324872    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:49:05.325929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.325871    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 11:49:05.580732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.580638    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" event={"ID":"72f8c531-d78b-45ac-a252-dec3ba7d0cc5","Type":"ContainerStarted","Data":"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"}
Oct 30 11:49:05.580732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.580689    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" event={"ID":"72f8c531-d78b-45ac-a252-dec3ba7d0cc5","Type":"ContainerStarted","Data":"6c46726c947207a6a80c37fe99486b74b625d14ea8d00d3b66d62d7bb913f844"}
Oct 30 11:49:05.620427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.620320    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" podStartSLOduration=1.620298353 podStartE2EDuration="1.620298353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:05.618517193 +0000 UTC m=+1144.838802104" watchObservedRunningTime="2026-10-30 11:49:05.620298353 +0000 UTC m=+1144.840583263"
Oct 30 11:49:05.756131 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:05.756098    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:07.575702 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:49:07.575668    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:49:13.689505 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:13.689458    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.221758 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:14.221711    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.341522 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:14.341474    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f" podUID="7843c47a-0b1d-4004-a852-923b462f0c5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:49:14.778350 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:14.777282    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:14.861025 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:14.860991    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:15.763528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.763498    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:49:15.866355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.865443    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 11:49:15.894453 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.882038    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.937042 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.929287    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 11:49:15.972356 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.951527    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.972767 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.972737    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.972973 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.972945    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.973140 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.973112    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjjk\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.973667 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.973313    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.981638 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.981615    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.981878 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.981841    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.982056 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.982031    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:15.982271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:15.982251    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.084944    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085051    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085167    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085267    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085302    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085356    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjjk\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085400    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085446    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085481    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.085906    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.086696 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.086390    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.110672 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.110638    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.110875 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.110855    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.117712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.117655    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.124736 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.120223    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.127829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.127788    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.135585 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.131055    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjjk\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.153292 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.151880    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.262956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.262918    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:16.593016 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:16.592987    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dc15c7_58bb_475f_9f1c_11b0fa049309.slice/crio-7f80bf9291f1255dd79fdf4a30eb246aa2bfaa44f1413aa3f60f033547961445 WatchSource:0}: Error finding container 7f80bf9291f1255dd79fdf4a30eb246aa2bfaa44f1413aa3f60f033547961445: Status 404 returned error can't find the container with id 7f80bf9291f1255dd79fdf4a30eb246aa2bfaa44f1413aa3f60f033547961445
Oct 30 11:49:16.594101 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.594073    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 11:49:16.595647 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.595610    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:16.595737 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.595692    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:16.595783 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.595741    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:16.685190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.685139    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" event={"ID":"80dc15c7-58bb-475f-9f1c-11b0fa049309","Type":"ContainerStarted","Data":"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"}
Oct 30 11:49:16.685370 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.685212    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" event={"ID":"80dc15c7-58bb-475f-9f1c-11b0fa049309","Type":"ContainerStarted","Data":"7f80bf9291f1255dd79fdf4a30eb246aa2bfaa44f1413aa3f60f033547961445"}
Oct 30 11:49:16.728038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:16.727985    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podStartSLOduration=1.727966721 podStartE2EDuration="1.727966721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:16.725979177 +0000 UTC m=+1155.946264087" watchObservedRunningTime="2026-10-30 11:49:16.727966721 +0000 UTC m=+1155.948251633"
Oct 30 11:49:17.274394 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:17.266104    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:17.277678 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:17.277492    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:17.277678 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:17.277556    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:18.264376 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.264321    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:18.264555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.264403    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:18.637405 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.637371    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:49:18.641871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.641847    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.656421 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.656400    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-71af6f-dockercfg-wpvtv\""
Oct 30 11:49:18.667545 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.667521    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:49:18.747148 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.747111    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.747519 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.747495    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwt52\" (UniqueName: \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.748984 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.748965    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71af6f\""
Oct 30 11:49:18.852357 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.852298    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.852551 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.852372    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwt52\" (UniqueName: \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.853642 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.853613    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.887383 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.883491    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwt52\" (UniqueName: \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\") pod \"payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:18.957746 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:18.957678    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:19.263613 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.263537    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:19.263613 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.263586    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:19.318771 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.318742    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:49:19.320664 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:19.320635    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee60170_0013_4563_8f85_392d03e6e0a7.slice/crio-a178843cdf19d3457fadd6413220e1e6fe4728b1fb79869c7b7373a8f80e6cb3 WatchSource:0}: Error finding container a178843cdf19d3457fadd6413220e1e6fe4728b1fb79869c7b7373a8f80e6cb3: Status 404 returned error can't find the container with id a178843cdf19d3457fadd6413220e1e6fe4728b1fb79869c7b7373a8f80e6cb3
Oct 30 11:49:19.740102 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.740063    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" event={"ID":"5ee60170-0013-4563-8f85-392d03e6e0a7","Type":"ContainerStarted","Data":"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"}
Oct 30 11:49:19.740607 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.740137    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:19.740607 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.740156    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" event={"ID":"5ee60170-0013-4563-8f85-392d03e6e0a7","Type":"ContainerStarted","Data":"a178843cdf19d3457fadd6413220e1e6fe4728b1fb79869c7b7373a8f80e6cb3"}
Oct 30 11:49:19.775820 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:19.775744    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" podStartSLOduration=1.77573239 podStartE2EDuration="1.77573239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:19.775722374 +0000 UTC m=+1158.996007283" watchObservedRunningTime="2026-10-30 11:49:19.77573239 +0000 UTC m=+1158.996017299"
Oct 30 11:49:20.263478 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:20.263444    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:20.263630 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:20.263496    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:21.264190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:21.264155    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:21.264587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:21.264207    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:22.263963 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:22.263926    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:22.264140 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:22.263976    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:23.263723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:23.263692    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 11:49:23.264114 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:23.263741    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 11:49:24.267724 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:24.267694    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:24.268217 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:24.268172    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:24.268800 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:24.268779    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 11:49:30.751784 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.751752    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:49:30.821871 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.821820    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 11:49:30.826149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.826125    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.848697 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.848659    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 11:49:30.936223 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936187    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936443 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936247    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936443 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936355    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936443 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936422    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936638 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936543    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936638 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936581    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936638 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936607    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4s5k\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936635    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:30.936741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:30.936686    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038310 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038233    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038310 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038298    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038322    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038393    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038501    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038526    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038549    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4s5k\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038606 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038581    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038606    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038722    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.038935 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.038837    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.039166 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.039140    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.039409 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.039383    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.039479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.039436    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.041038 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.041008    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.041163 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.041144    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.059798 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.059764    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.059923 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.059859    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4s5k\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\") pod \"e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") " pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.141712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.141680    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:31.458299 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.458270    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 11:49:31.458910 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:31.458884    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f7574c_e999_4d5b_b6f1_68b5126dfff4.slice/crio-9d8dd8086cd1d44815536ba067b0ed166645376edc500249698c2ef00a05a8b0 WatchSource:0}: Error finding container 9d8dd8086cd1d44815536ba067b0ed166645376edc500249698c2ef00a05a8b0: Status 404 returned error can't find the container with id 9d8dd8086cd1d44815536ba067b0ed166645376edc500249698c2ef00a05a8b0
Oct 30 11:49:31.461403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.461367    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:31.461495 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.461434    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:31.461495 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.461466    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:31.818889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.818796    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" event={"ID":"38f7574c-e999-4d5b-b6f1-68b5126dfff4","Type":"ContainerStarted","Data":"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"}
Oct 30 11:49:31.818889 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.818865    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" event={"ID":"38f7574c-e999-4d5b-b6f1-68b5126dfff4","Type":"ContainerStarted","Data":"9d8dd8086cd1d44815536ba067b0ed166645376edc500249698c2ef00a05a8b0"}
Oct 30 11:49:31.862116 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:31.862063    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podStartSLOduration=1.862049463 podStartE2EDuration="1.862049463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:31.858695103 +0000 UTC m=+1171.078980031" watchObservedRunningTime="2026-10-30 11:49:31.862049463 +0000 UTC m=+1171.082334374"
Oct 30 11:49:32.142340 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:32.142294    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:32.143883 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:32.143855    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:32.144012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:32.143903    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:33.149355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:33.148741    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:33.149355 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:33.148812    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:34.141990 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:34.141954    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:34.142180 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:34.142008    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:35.142441 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.142406    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:35.142832 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.142461    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:35.614982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.614938    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"]
Oct 30 11:49:35.620294 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.620269    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.645121 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.645094    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 11:49:35.645295 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.645097    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 11:49:35.645295 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.645107    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 11:49:35.645515 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.645123    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dn7s8\""
Oct 30 11:49:35.651216 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.651186    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"]
Oct 30 11:49:35.723893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.723863    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724069 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.723995    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724069 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.724046    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrmf\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-kube-api-access-fkrmf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.724076    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e04985-dc6f-43e4-ba67-2ae07d896050-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724265 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.724242    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724307 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.724289    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.724387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.724367    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826071 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826030    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826093    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrmf\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-kube-api-access-fkrmf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826125    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e04985-dc6f-43e4-ba67-2ae07d896050-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826210    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826254    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826285 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826286    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.826640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.826414    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.827255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.827227    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.828789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.828762    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/32e04985-dc6f-43e4-ba67-2ae07d896050-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.828992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.828972    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.829158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.829116    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.829477 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.829456    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.848308 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.848280    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.848504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.848479    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrmf\" (UniqueName: \"kubernetes.io/projected/32e04985-dc6f-43e4-ba67-2ae07d896050-kube-api-access-fkrmf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7htsq\" (UID: \"32e04985-dc6f-43e4-ba67-2ae07d896050\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:35.934316 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:35.934234    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:36.142861 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.142829    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:36.143217 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.142890    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:36.315540 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.315440    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"]
Oct 30 11:49:36.320292 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:49:36.320254    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e04985_dc6f_43e4_ba67_2ae07d896050.slice/crio-ae57d7d19f329aab20afc915fc20f223673153322acf44022c7c83ce0ae8a467 WatchSource:0}: Error finding container ae57d7d19f329aab20afc915fc20f223673153322acf44022c7c83ce0ae8a467: Status 404 returned error can't find the container with id ae57d7d19f329aab20afc915fc20f223673153322acf44022c7c83ce0ae8a467
Oct 30 11:49:36.322887 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.322851    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:36.322966 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.322916    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:36.865508 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.865467    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq" event={"ID":"32e04985-dc6f-43e4-ba67-2ae07d896050","Type":"ContainerStarted","Data":"0da9b5578fae8b3c980d4c007173695f69738405ad558ec3fbd4ab115f5feef8"}
Oct 30 11:49:36.865687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.865525    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq" event={"ID":"32e04985-dc6f-43e4-ba67-2ae07d896050","Type":"ContainerStarted","Data":"ae57d7d19f329aab20afc915fc20f223673153322acf44022c7c83ce0ae8a467"}
Oct 30 11:49:36.865687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.865566    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:36.916375 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:36.916056    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq" podStartSLOduration=1.916037875 podStartE2EDuration="1.916037875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:49:36.905778421 +0000 UTC m=+1176.126063332" watchObservedRunningTime="2026-10-30 11:49:36.916037875 +0000 UTC m=+1176.136322786"
Oct 30 11:49:37.142704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:37.142617    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:37.142704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:37.142677    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:37.576637 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:49:37.576553    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:49:37.873850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:37.873813    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7htsq"
Oct 30 11:49:38.142327 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:38.142242    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 11:49:38.142327 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:38.142298    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 11:49:39.146263 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:39.146229    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:39.146742 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:39.146721    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:39.147460 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:39.147439    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 11:49:55.046701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.046594    4453 generic.go:356] "Generic (PLEG): container finished" podID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerID="79ebf17442c084bd23a536bb52c8473c6e5f872315d84edf2201db812232e84c" exitCode=0
Oct 30 11:49:55.046701 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.046671    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerDied","Data":"79ebf17442c084bd23a536bb52c8473c6e5f872315d84edf2201db812232e84c"}
Oct 30 11:49:55.047319 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.047244    4453 scope.go:122] "RemoveContainer" containerID="79ebf17442c084bd23a536bb52c8473c6e5f872315d84edf2201db812232e84c"
Oct 30 11:49:55.049073 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.049040    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:55.049182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.049113    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:55.049182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:55.049145    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:49:56.136422 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:56.136371    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerStarted","Data":"4432154726f2e91398d4e2fcd3b54d9f842ba8065e257fe14a2423740c0ffaec"}
Oct 30 11:49:56.332174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:56.332139    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:49:56.332461 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:56.332431    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:49:56.332538 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:56.332494    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:49:57.326541 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:57.326505    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:49:57.327002 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:57.326563    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:49:58.327065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:58.327033    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:49:58.327475 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:58.327092    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:49:59.326721 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:59.326684    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:49:59.326899 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:49:59.326745    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:50:00.327026 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:00.326987    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:50:00.327646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:00.327460    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:50:01.327360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:01.327299    4453 patch_prober.go:29] interesting pod/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 11:50:01.327778 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:01.327383    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 11:50:02.331712 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:02.331674    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:50:02.332114 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:02.332065    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:50:02.332660 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:50:02.332640    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:50:07.570400 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:50:07.570321    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:50:37.570866 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:50:37.570825    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:51:07.571924 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:51:07.571892    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:51:26.265449 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:26.265416    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:51:31.266003 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.265962    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" podUID="5ee60170-0013-4563-8f85-392d03e6e0a7" containerName="payload-pre-processing" containerID="cri-o://bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051" gracePeriod=25
Oct 30 11:51:31.558009 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.557980    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:51:31.642970 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.642942    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\" (UniqueName: \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\") pod \"5ee60170-0013-4563-8f85-392d03e6e0a7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") "
Oct 30 11:51:31.643158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.642999    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\") pod \"5ee60170-0013-4563-8f85-392d03e6e0a7\" (UID: \"5ee60170-0013-4563-8f85-392d03e6e0a7\") "
Oct 30 11:51:31.643433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.643406    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume" pod "5ee60170-0013-4563-8f85-392d03e6e0a7" (UID: "5ee60170-0013-4563-8f85-392d03e6e0a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:51:31.645031 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.645002    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52" pod "5ee60170-0013-4563-8f85-392d03e6e0a7" (UID: "5ee60170-0013-4563-8f85-392d03e6e0a7"). InnerVolumeSpecName "kube-api-access-dwt52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:31.744360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.744303    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwt52\" (UniqueName: \"kubernetes.io/projected/5ee60170-0013-4563-8f85-392d03e6e0a7-kube-api-access-dwt52\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:31.744520 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.744366    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee60170-0013-4563-8f85-392d03e6e0a7-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:31.923554 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.923519    4453 generic.go:356] "Generic (PLEG): container finished" podID="5ee60170-0013-4563-8f85-392d03e6e0a7" containerID="bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051" exitCode=0
Oct 30 11:51:31.923759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.923569    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"
Oct 30 11:51:31.923759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.923606    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" event={"ID":"5ee60170-0013-4563-8f85-392d03e6e0a7","Type":"ContainerDied","Data":"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"}
Oct 30 11:51:31.923759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.923671    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7" event={"ID":"5ee60170-0013-4563-8f85-392d03e6e0a7","Type":"ContainerDied","Data":"a178843cdf19d3457fadd6413220e1e6fe4728b1fb79869c7b7373a8f80e6cb3"}
Oct 30 11:51:31.923759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.923704    4453 scope.go:122] "RemoveContainer" containerID="bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"
Oct 30 11:51:31.933647 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.933626    4453 scope.go:122] "RemoveContainer" containerID="bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"
Oct 30 11:51:31.933954 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:51:31.933935    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051\": container with ID starting with bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051 not found: ID does not exist" containerID="bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"
Oct 30 11:51:31.934034 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.933961    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051"} err="failed to get container status \"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051\": rpc error: code = NotFound desc = could not find container \"bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051\": container with ID starting with bac439155521f33f2ccd4a1072587ff5f2cd365083db2298c9535a08a2d40051 not found: ID does not exist"
Oct 30 11:51:31.972275 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.972243    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:51:31.976780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:31.976749    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71af6f-fc55b68f9-k9xn7"]
Oct 30 11:51:32.913896 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:32.913864    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee60170-0013-4563-8f85-392d03e6e0a7" path="/var/lib/kubelet/pods/5ee60170-0013-4563-8f85-392d03e6e0a7/volumes"
Oct 30 11:51:37.571346 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:51:37.571296    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:51:51.701220 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:51.701110    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:51:51.701800 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:51.701438    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy" containerID="cri-o://4432154726f2e91398d4e2fcd3b54d9f842ba8065e257fe14a2423740c0ffaec" gracePeriod=30
Oct 30 11:51:57.091986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.091905    4453 generic.go:356] "Generic (PLEG): container finished" podID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerID="4432154726f2e91398d4e2fcd3b54d9f842ba8065e257fe14a2423740c0ffaec" exitCode=0
Oct 30 11:51:57.091986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.091970    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerDied","Data":"4432154726f2e91398d4e2fcd3b54d9f842ba8065e257fe14a2423740c0ffaec"}
Oct 30 11:51:57.092414 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.092031    4453 scope.go:122] "RemoveContainer" containerID="79ebf17442c084bd23a536bb52c8473c6e5f872315d84edf2201db812232e84c"
Oct 30 11:51:57.376203 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.376176    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:51:57.507115 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507080    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507160    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507322 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507194    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507489 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507439    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507489 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507429    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:51:57.507604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507523    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507604 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507568    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507710 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507610    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507710 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507677    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.507820 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507733    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:51:57.507820 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507766    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\") pod \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\" (UID: \"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1\") "
Oct 30 11:51:57.508081 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.507998    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:51:57.508081 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508034    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:51:57.508403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508377    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.508528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508405    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.508528 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508520    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.508653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508538    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.508653 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.508518    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:51:57.511440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.511412    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:51:57.511692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.511664    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "kube-api-access-56pw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:57.511816 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.511782    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:51:57.512428 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.512408    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy" pod "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" (UID: "7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:51:57.610057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.610023    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.610057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.610053    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.610234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.610062    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.610234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.610071    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:57.610234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:57.610080    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1-kube-api-access-56pw4\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:51:58.099986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.099893    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn" event={"ID":"7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1","Type":"ContainerDied","Data":"8497877b4cebd794d3372f657237b1e7c5a99569d955ca4b14b448d5fb410ac8"}
Oct 30 11:51:58.099986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.099918    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"
Oct 30 11:51:58.099986 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.099963    4453 scope.go:122] "RemoveContainer" containerID="4432154726f2e91398d4e2fcd3b54d9f842ba8065e257fe14a2423740c0ffaec"
Oct 30 11:51:58.166549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.166516    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:51:58.171933 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.171905    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-71af6f-openshift-default-795bcf9cf-qjcwn"]
Oct 30 11:51:58.912074 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:51:58.912038    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" path="/var/lib/kubelet/pods/7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1/volumes"
Oct 30 11:52:07.571223 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:52:07.571178    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:52:23.412371 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.412313    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:52:23.413185 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413163    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413236 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413185    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413236 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413214    4453 cpu_manager.go:414] "containerMap: removing container" podUID="5ee60170-0013-4563-8f85-392d03e6e0a7" containerName="payload-pre-processing"
Oct 30 11:52:23.413236 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413219    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee60170-0013-4563-8f85-392d03e6e0a7" containerName="payload-pre-processing"
Oct 30 11:52:23.413351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413238    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413245    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413426 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413348    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413426 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413390    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d1eaf45-4d7c-4ceb-8019-0707cfcfb6b1" containerName="istio-proxy"
Oct 30 11:52:23.413426 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.413415    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee60170-0013-4563-8f85-392d03e6e0a7" containerName="payload-pre-processing"
Oct 30 11:52:23.417010 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.416987    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.431605 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.431587    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-caa7a0-serving-cert\""
Oct 30 11:52:23.445295 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.445271    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:52:23.498394 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.498355    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.498549 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.498497    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.498621 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.498599    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv8r\" (UniqueName: \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.498666 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.498650    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.599944 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.599909    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.600141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.599980    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv8r\" (UniqueName: \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.600141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.600018    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.600141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.600108    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.600662 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.600636    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.602445 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.602417    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.602445 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.602440    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.647455 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.638261    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv8r\" (UniqueName: \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\") pod \"maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:23.731349 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:23.731244    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:24.066279 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:24.066249    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:52:24.067509 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:52:24.067477    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7af338e_a969_49e5_8468_25754331e968.slice/crio-5238f1c0eb21bca48ae259fa9f326a34aed584d9146a4a7dcec3bdfa76d3d723 WatchSource:0}: Error finding container 5238f1c0eb21bca48ae259fa9f326a34aed584d9146a4a7dcec3bdfa76d3d723: Status 404 returned error can't find the container with id 5238f1c0eb21bca48ae259fa9f326a34aed584d9146a4a7dcec3bdfa76d3d723
Oct 30 11:52:24.068927 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:24.068904    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 11:52:24.381922 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:24.381882    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerStarted","Data":"5238f1c0eb21bca48ae259fa9f326a34aed584d9146a4a7dcec3bdfa76d3d723"}
Oct 30 11:52:25.420956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:25.413781    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerStarted","Data":"64949a44bf2048fd4fd01783316c777fb601e61d9631ed4d04b132258cc4d03a"}
Oct 30 11:52:25.420956 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:25.414912    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:25.455611 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:25.455555    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podStartSLOduration=2.0468906 podStartE2EDuration="2.455539326s" totalImagesPullingTime="408.648726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:52:25.452898896 +0000 UTC m=+1344.673183806" watchObservedRunningTime="2026-10-30 11:52:25.455539326 +0000 UTC m=+1344.675824260"
Oct 30 11:52:32.427751 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:32.427704    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": dial tcp 10.133.0.84:8443: connect: connection refused"
Oct 30 11:52:35.476923 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:35.476885    4453 generic.go:356] "Generic (PLEG): container finished" podID="a7af338e-a969-49e5-8468-25754331e968" containerID="64949a44bf2048fd4fd01783316c777fb601e61d9631ed4d04b132258cc4d03a" exitCode=1
Oct 30 11:52:35.477428 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:35.476956    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerDied","Data":"64949a44bf2048fd4fd01783316c777fb601e61d9631ed4d04b132258cc4d03a"}
Oct 30 11:52:35.477575 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:35.477557    4453 scope.go:122] "RemoveContainer" containerID="64949a44bf2048fd4fd01783316c777fb601e61d9631ed4d04b132258cc4d03a"
Oct 30 11:52:36.484966 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:36.484929    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerStarted","Data":"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"}
Oct 30 11:52:36.485440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:36.485151    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:37.572203 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:52:37.572166    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:52:42.493519 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:42.493478    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.84:8443/health\": dial tcp 10.133.0.84:8443: connect: connection refused"
Oct 30 11:52:46.561105 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:46.561069    4453 generic.go:356] "Generic (PLEG): container finished" podID="a7af338e-a969-49e5-8468-25754331e968" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e" exitCode=1
Oct 30 11:52:46.561523 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:46.561180    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerDied","Data":"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"}
Oct 30 11:52:46.561523 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:46.561222    4453 scope.go:122] "RemoveContainer" containerID="64949a44bf2048fd4fd01783316c777fb601e61d9631ed4d04b132258cc4d03a"
Oct 30 11:52:46.561703 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:46.561686    4453 scope.go:122] "RemoveContainer" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:52:46.562001 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:52:46.561976    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb_odh-ai-gateway-infra(a7af338e-a969-49e5-8468-25754331e968)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podUID="a7af338e-a969-49e5-8468-25754331e968"
Oct 30 11:52:53.731724 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:53.731690    4453 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:52:53.732225 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:53.732203    4453 scope.go:122] "RemoveContainer" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:52:53.732558 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:52:53.732537    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb_odh-ai-gateway-infra(a7af338e-a969-49e5-8468-25754331e968)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podUID="a7af338e-a969-49e5-8468-25754331e968"
Oct 30 11:52:56.905303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:56.905271    4453 scope.go:122] "RemoveContainer" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:52:57.629600 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:57.629567    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerStarted","Data":"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"}
Oct 30 11:52:57.630366 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:52:57.630321    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:53:03.641574 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:03.641543    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:53:07.583317 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:53:07.583267    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:53:12.448725 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.448692    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:53:12.452902 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.452879    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.471574 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.471545    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6d9258eb-openshift-default-dockercfg-tx4ln\""
Oct 30 11:53:12.485231 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.485199    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:53:12.524638 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.524584    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbsl\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.524833 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.524651    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.524833 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.524776    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.524989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.524906    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.524989 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.524966    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.525098 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.525037    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.525153 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.525108    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.525234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.525214    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.525378 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.525249    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.625976 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.625941    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbsl\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626127 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.625988    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626127 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626110    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626222    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626256    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626363 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626316    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626440 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626399    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626582 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626553    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626738 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626609    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626738 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626691    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626853 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626836    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.626965 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.626941    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.627035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.627019    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.627258 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.627236    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.628601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.628576    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.629039 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.629021    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.646596 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.646567    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.646898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.646877    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbsl\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:12.782065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:12.781985    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:13.112255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.112227    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:53:13.121723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.121686    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:13.121827 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.121776    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:13.121891 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.121826    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:13.760865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.760786    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" event={"ID":"8d029cc9-7523-4acb-b82d-cdabd1bad5a6","Type":"ContainerStarted","Data":"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"}
Oct 30 11:53:13.761269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.760867    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" event={"ID":"8d029cc9-7523-4acb-b82d-cdabd1bad5a6","Type":"ContainerStarted","Data":"c7b96b140207f03d4200e658ffe22c8615c317de9f1c53f4b3ad425bd8044480"}
Oct 30 11:53:13.782236 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.782206    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:13.811429 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:13.811371    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podStartSLOduration=1.811325356 podStartE2EDuration="1.811325356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:13.80869654 +0000 UTC m=+1393.028981450" watchObservedRunningTime="2026-10-30 11:53:13.811325356 +0000 UTC m=+1393.031610268"
Oct 30 11:53:14.784264 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:14.784228    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:14.784734 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:14.784286    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:15.783382 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:15.783352    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:15.783555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:15.783424    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:16.783077 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:16.783049    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:16.783501 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:16.783096    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:17.786680 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:17.786643    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:17.799045 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:17.799004    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:18.782635 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:18.782597    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:18.782815 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:18.782670    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:19.782943 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:19.782901    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 11:53:19.783365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:19.782972    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 11:53:20.789458 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:20.789428    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:20.789972 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:20.789949    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:20.790785 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:20.790761    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:53:23.956179 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:23.956146    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:53:27.497925 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.497887    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:53:27.502287 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.502264    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.530929 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.530896    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:53:27.551446 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.551408    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.551665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.551604    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.551777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.551708    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.551777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.551764    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.551921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.551803    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgp9t\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.552695 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.552670    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.552903 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.552884    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.553047 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.553001    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.553141 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.553046    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.654823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654789    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.654823 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654829    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654853    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgp9t\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654892    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654946    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.654993    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655009    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655320 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655080    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655320 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655184    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655320 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655251    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655320 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655301    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655647 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655626    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655659    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.655828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.655811    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.657179 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.657161    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.657491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.657473    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.688702 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.688417    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.691723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.689834    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgp9t\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:27.819713 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:27.819599    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:28.140020 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.139985    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:53:28.140949 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:53:28.140921    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb734c3_9bca_4e41_82ad_3bd9575507dc.slice/crio-bb6a2e9fe95ffb89e20cfbc6059ba98de99702cb3966353bd11233ee4c11ae66 WatchSource:0}: Error finding container bb6a2e9fe95ffb89e20cfbc6059ba98de99702cb3966353bd11233ee4c11ae66: Status 404 returned error can't find the container with id bb6a2e9fe95ffb89e20cfbc6059ba98de99702cb3966353bd11233ee4c11ae66
Oct 30 11:53:28.143903 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.143859    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:28.144012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.143953    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:28.144012 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.144002    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:28.920888 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.920848    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" event={"ID":"8fb734c3-9bca-4e41-82ad-3bd9575507dc","Type":"ContainerStarted","Data":"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"}
Oct 30 11:53:28.921279 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.920900    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" event={"ID":"8fb734c3-9bca-4e41-82ad-3bd9575507dc","Type":"ContainerStarted","Data":"bb6a2e9fe95ffb89e20cfbc6059ba98de99702cb3966353bd11233ee4c11ae66"}
Oct 30 11:53:28.981529 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:28.981482    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podStartSLOduration=1.98146806 podStartE2EDuration="1.98146806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:28.969554472 +0000 UTC m=+1408.189839405" watchObservedRunningTime="2026-10-30 11:53:28.98146806 +0000 UTC m=+1408.201752971"
Oct 30 11:53:29.819992 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:29.819956    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:29.821828 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:29.821796    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:53:29.821952 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:29.821847    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 11:53:30.820978 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:30.820941    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:53:30.821405 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:30.821006    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 11:53:31.820666 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:31.820625    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:53:31.820836 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:31.820677    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 11:53:32.820061 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:32.820026    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:53:32.820584 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:32.820555    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 11:53:33.826360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:33.826135    4453 patch_prober.go:29] interesting pod/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 11:53:33.826360 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:33.826204    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 11:53:34.825431 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:34.825401    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:34.825873 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:34.825855    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:34.827006 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:34.826982    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:53:37.572942 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:53:37.572900    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:53:38.977449 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:38.977392    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" podUID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" containerName="main" containerID="cri-o://221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b" gracePeriod=45
Oct 30 11:53:39.414471 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.414443    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:53:39.573266 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573171    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573271    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573459 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573301    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573478    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573532    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:39.573587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573551    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\" (UniqueName: \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573587 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573569    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:39.573792 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573657    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\") pod \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\" (UID: \"72f8c531-d78b-45ac-a252-dec3ba7d0cc5\") "
Oct 30 11:53:39.573792 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.573733    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:39.574219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.574199    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:39.574219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.574218    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:39.574444 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.574227    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:39.575892 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.575868    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:53:39.576115 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.576096    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:39.576174 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.576154    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb" pod "72f8c531-d78b-45ac-a252-dec3ba7d0cc5" (UID: "72f8c531-d78b-45ac-a252-dec3ba7d0cc5"). InnerVolumeSpecName "kube-api-access-ntwgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:39.675149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.675108    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:39.675149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.675142    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntwgb\" (UniqueName: \"kubernetes.io/projected/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-kube-api-access-ntwgb\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:39.675149 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:39.675153    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f8c531-d78b-45ac-a252-dec3ba7d0cc5-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:40.127257 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.127223    4453 generic.go:356] "Generic (PLEG): container finished" podID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" containerID="221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b" exitCode=0
Oct 30 11:53:40.127731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.127276    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"
Oct 30 11:53:40.127731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.127310    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" event={"ID":"72f8c531-d78b-45ac-a252-dec3ba7d0cc5","Type":"ContainerDied","Data":"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"}
Oct 30 11:53:40.127731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.127370    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5" event={"ID":"72f8c531-d78b-45ac-a252-dec3ba7d0cc5","Type":"ContainerDied","Data":"6c46726c947207a6a80c37fe99486b74b625d14ea8d00d3b66d62d7bb913f844"}
Oct 30 11:53:40.127731 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.127387    4453 scope.go:122] "RemoveContainer" containerID="221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"
Oct 30 11:53:40.146316 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.146294    4453 scope.go:122] "RemoveContainer" containerID="221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"
Oct 30 11:53:40.148652 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:53:40.148617    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b\": container with ID starting with 221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b not found: ID does not exist" containerID="221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"
Oct 30 11:53:40.148768 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.148664    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b"} err="failed to get container status \"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b\": rpc error: code = NotFound desc = could not find container \"221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b\": container with ID starting with 221f8a18bfda9a2228b6a83c7f190c1150aa2c335054616600e01fdbfd79607b not found: ID does not exist"
Oct 30 11:53:40.182739 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.182709    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:53:40.198365 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.190408    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6d9258eb/e2e-lifecycle-model-6d9258eb-kserve-764f8c86f-wjjb5"]
Oct 30 11:53:40.314255 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.314220    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:53:40.314748 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.314715    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy" containerID="cri-o://40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12" gracePeriod=30
Oct 30 11:53:40.920569 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:40.920533    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" path="/var/lib/kubelet/pods/72f8c531-d78b-45ac-a252-dec3ba7d0cc5/volumes"
Oct 30 11:53:42.526640 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.526607    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:53:42.527236 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.527220    4453 cpu_manager.go:414] "containerMap: removing container" podUID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" containerName="main"
Oct 30 11:53:42.527280 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.527237    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" containerName="main"
Oct 30 11:53:42.527369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.527359    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f8c531-d78b-45ac-a252-dec3ba7d0cc5" containerName="main"
Oct 30 11:53:42.531993 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.531972    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.560523 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.560488    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:53:42.721187 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721151    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721187 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721187    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721292    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721406 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721379    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721473 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721438    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721466    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7z6\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721544 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721506    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721580 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721553    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.721732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.721718    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823370 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823272    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823541 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823386    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823541 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823414    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823541 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823470    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823541 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823501    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823575    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823599    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7z6\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823624    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.823788 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823651    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.824011 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823895    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.824011 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823920    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.824011 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.823975    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.824209 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.824187    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.824367 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.824349    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.825631 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.825616    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.825985 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.825966    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.843601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.843574    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.843705 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.843636    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7z6\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\") pod \"e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") " pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:42.846483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:42.846466    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:43.167805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:43.167777    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:53:43.169000 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:53:43.168970    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c219d8_565c_4059_9365_0d19a1ec8a3b.slice/crio-a69dc1f1f478e9aaac21892ec743de0fd07ed29adbbe0e2ccbea1fe5f7968524 WatchSource:0}: Error finding container a69dc1f1f478e9aaac21892ec743de0fd07ed29adbbe0e2ccbea1fe5f7968524: Status 404 returned error can't find the container with id a69dc1f1f478e9aaac21892ec743de0fd07ed29adbbe0e2ccbea1fe5f7968524
Oct 30 11:53:43.171479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:43.171447    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:43.171568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:43.171514    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:43.171568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:43.171546    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:53:44.155704 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:44.155662    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" event={"ID":"94c219d8-565c-4059-9365-0d19a1ec8a3b","Type":"ContainerStarted","Data":"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"}
Oct 30 11:53:44.156223 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:44.155740    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" event={"ID":"94c219d8-565c-4059-9365-0d19a1ec8a3b","Type":"ContainerStarted","Data":"a69dc1f1f478e9aaac21892ec743de0fd07ed29adbbe0e2ccbea1fe5f7968524"}
Oct 30 11:53:44.278854 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:44.278781    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" podStartSLOduration=2.278760599 podStartE2EDuration="2.278760599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:53:44.275766593 +0000 UTC m=+1423.496051504" watchObservedRunningTime="2026-10-30 11:53:44.278760599 +0000 UTC m=+1423.499045510"
Oct 30 11:53:44.847165 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:44.847124    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:44.855184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:44.855146    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:45.162067 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:45.162033    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:45.163057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:45.163036    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:53:46.015239 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.015209    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:53:46.106242 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106213    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106438 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106283    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106438 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106387    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106470    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106520    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106561    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:46.106673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106582    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106673 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106661    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106825 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106719    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106825 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106754    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\") pod \"491fc23e-5170-4eba-af16-371e67138787\" (UID: \"491fc23e-5170-4eba-af16-371e67138787\") "
Oct 30 11:53:46.106825 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.106787    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:46.107387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107015    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:46.107387 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107211    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:46.107634 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107577    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.107634 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107604    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.107634 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107622    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.107842 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.107636    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.108589 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.108565    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:53:46.109491 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.109466    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:46.111539 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.111400    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:53:46.111539 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.111506    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "kube-api-access-22bxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:53:46.111821 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.111794    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo" pod "491fc23e-5170-4eba-af16-371e67138787" (UID: "491fc23e-5170-4eba-af16-371e67138787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:53:46.168759 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.168726    4453 generic.go:356] "Generic (PLEG): container finished" podID="491fc23e-5170-4eba-af16-371e67138787" containerID="40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12" exitCode=0
Oct 30 11:53:46.169252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.168771    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"
Oct 30 11:53:46.169252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.168811    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" event={"ID":"491fc23e-5170-4eba-af16-371e67138787","Type":"ContainerDied","Data":"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"}
Oct 30 11:53:46.169252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.168876    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs" event={"ID":"491fc23e-5170-4eba-af16-371e67138787","Type":"ContainerDied","Data":"702dc883fcaeb327df371eecb13d820069b5845b6832f5abb07cc7b21a32cf96"}
Oct 30 11:53:46.169252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.168901    4453 scope.go:122] "RemoveContainer" containerID="40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"
Oct 30 11:53:46.189133 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.189106    4453 scope.go:122] "RemoveContainer" containerID="40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"
Oct 30 11:53:46.189655 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:53:46.189602    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12\": container with ID starting with 40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12 not found: ID does not exist" containerID="40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"
Oct 30 11:53:46.189655 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.189636    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12"} err="failed to get container status \"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12\": rpc error: code = NotFound desc = could not find container \"40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12\": container with ID starting with 40fa7bedd445a5a66bac5c8c73ca55fcfafa66ab82d84c4d1fb125c588240e12 not found: ID does not exist"
Oct 30 11:53:46.209040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.209013    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/491fc23e-5170-4eba-af16-371e67138787-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.209040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.209037    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/491fc23e-5170-4eba-af16-371e67138787-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.209040 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.209046    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.209277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.209061    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bxb\" (UniqueName: \"kubernetes.io/projected/491fc23e-5170-4eba-af16-371e67138787-kube-api-access-22bxb\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.209277 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.209073    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/491fc23e-5170-4eba-af16-371e67138787-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:53:46.223119 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.223082    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:53:46.227910 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.227885    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-v5xcs"]
Oct 30 11:53:46.912568 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:53:46.912535    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="491fc23e-5170-4eba-af16-371e67138787" path="/var/lib/kubelet/pods/491fc23e-5170-4eba-af16-371e67138787/volumes"
Oct 30 11:54:07.570270 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:07.570233    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:54:08.971646 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.971611    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:54:08.972168 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.971895    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy" containerID="cri-o://cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e" gracePeriod=30
Oct 30 11:54:08.980151 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.980089    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:54:08.980975 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.980517    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" podUID="94c219d8-565c-4059-9365-0d19a1ec8a3b" containerName="istio-proxy" containerID="cri-o://997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a" gracePeriod=30
Oct 30 11:54:08.999456 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.999426    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:54:08.999682 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:08.999658    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy" containerID="cri-o://8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5" gracePeriod=30
Oct 30 11:54:14.299368 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.299339    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:54:14.326412 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.326384    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:54:14.330559 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.330539    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:54:14.389748 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389713    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.389942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389773    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.389942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389815    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.389942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389853    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.389942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389889    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.390182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.389999    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.390182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390044    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.390182 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390098    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.390383 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390365    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\") pod \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\" (UID: \"8fb734c3-9bca-4e41-82ad-3bd9575507dc\") "
Oct 30 11:54:14.391028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390359    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.391028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390567    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.391028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390817    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.391028 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.390986    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.391628 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.391602    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.392269 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.392242    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.392601 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.392574    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.393262 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.393243    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "kube-api-access-vgp9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.393671 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.393649    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token" pod "8fb734c3-9bca-4e41-82ad-3bd9575507dc" (UID: "8fb734c3-9bca-4e41-82ad-3bd9575507dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.428377 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.428312    4453 generic.go:356] "Generic (PLEG): container finished" podID="94c219d8-565c-4059-9365-0d19a1ec8a3b" containerID="997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a" exitCode=0
Oct 30 11:54:14.428542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.428436    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"
Oct 30 11:54:14.428542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.428459    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" event={"ID":"94c219d8-565c-4059-9365-0d19a1ec8a3b","Type":"ContainerDied","Data":"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"}
Oct 30 11:54:14.428542 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.428510    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx" event={"ID":"94c219d8-565c-4059-9365-0d19a1ec8a3b","Type":"ContainerDied","Data":"a69dc1f1f478e9aaac21892ec743de0fd07ed29adbbe0e2ccbea1fe5f7968524"}
Oct 30 11:54:14.429016 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.428991    4453 scope.go:122] "RemoveContainer" containerID="997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"
Oct 30 11:54:14.430913 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.430845    4453 generic.go:356] "Generic (PLEG): container finished" podID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerID="8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5" exitCode=0
Oct 30 11:54:14.431021 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.431007    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"
Oct 30 11:54:14.431150 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.431133    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" event={"ID":"8d029cc9-7523-4acb-b82d-cdabd1bad5a6","Type":"ContainerDied","Data":"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"}
Oct 30 11:54:14.431213 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.431169    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9" event={"ID":"8d029cc9-7523-4acb-b82d-cdabd1bad5a6","Type":"ContainerDied","Data":"c7b96b140207f03d4200e658ffe22c8615c317de9f1c53f4b3ad425bd8044480"}
Oct 30 11:54:14.433374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.433308    4453 generic.go:356] "Generic (PLEG): container finished" podID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerID="cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e" exitCode=0
Oct 30 11:54:14.433374 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.433354    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" event={"ID":"8fb734c3-9bca-4e41-82ad-3bd9575507dc","Type":"ContainerDied","Data":"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"}
Oct 30 11:54:14.433565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.433399    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"
Oct 30 11:54:14.433565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.433404    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn" event={"ID":"8fb734c3-9bca-4e41-82ad-3bd9575507dc","Type":"ContainerDied","Data":"bb6a2e9fe95ffb89e20cfbc6059ba98de99702cb3966353bd11233ee4c11ae66"}
Oct 30 11:54:14.442847 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.442824    4453 scope.go:122] "RemoveContainer" containerID="997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"
Oct 30 11:54:14.443164 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:14.443142    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a\": container with ID starting with 997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a not found: ID does not exist" containerID="997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"
Oct 30 11:54:14.443272 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.443170    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a"} err="failed to get container status \"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a\": rpc error: code = NotFound desc = could not find container \"997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a\": container with ID starting with 997df0ae4708e6b2974ba75b921025b7c3a02617833192cd2f3c9842cfa7952a not found: ID does not exist"
Oct 30 11:54:14.443272 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.443188    4453 scope.go:122] "RemoveContainer" containerID="8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"
Oct 30 11:54:14.455219 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.455198    4453 scope.go:122] "RemoveContainer" containerID="8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"
Oct 30 11:54:14.455895 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:14.455865    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5\": container with ID starting with 8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5 not found: ID does not exist" containerID="8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"
Oct 30 11:54:14.455996 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.455908    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5"} err="failed to get container status \"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5\": rpc error: code = NotFound desc = could not find container \"8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5\": container with ID starting with 8327059192fcd8b10d110b5c6a7c673891ee032716119510680e14873ea325b5 not found: ID does not exist"
Oct 30 11:54:14.455996 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.455931    4453 scope.go:122] "RemoveContainer" containerID="cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"
Oct 30 11:54:14.469142 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.469125    4453 scope.go:122] "RemoveContainer" containerID="cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"
Oct 30 11:54:14.469660 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:14.469612    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e\": container with ID starting with cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e not found: ID does not exist" containerID="cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"
Oct 30 11:54:14.469769 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.469650    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e"} err="failed to get container status \"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e\": rpc error: code = NotFound desc = could not find container \"cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e\": container with ID starting with cd17772ae9d10cea435fc2ea0b30899c9a9e7aee92b47a92830a84243708216e not found: ID does not exist"
Oct 30 11:54:14.492265 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492234    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.492431 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492278    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.492431 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492365    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492433    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492466    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.492567 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492558    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492587    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492619    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.492732 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492645    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.492893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492739    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492799    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492827    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.492893 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492868    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.493107 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.492925    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.493222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.493151    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.493323 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.493235    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.493490 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.493467    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.493576 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.493525    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.494281 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.493448    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496277    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496326    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496393    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496418    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496442    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\") pod \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\" (UID: \"8d029cc9-7523-4acb-b82d-cdabd1bad5a6\") "
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496472    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\") pod \"94c219d8-565c-4059-9365-0d19a1ec8a3b\" (UID: \"94c219d8-565c-4059-9365-0d19a1ec8a3b\") "
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496760    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.496849    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.497014    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.497665 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.497388    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:14.499099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.499067    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.500937 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.500909    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgp9t\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-kube-api-access-vgp9t\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.500941    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.500955    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.500975    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.500989    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501002    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501014    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501031    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501043    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501055    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501067    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501085    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501098    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501111    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501123    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501139    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c219d8-565c-4059-9365-0d19a1ec8a3b-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501164    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501178    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501190    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.501503 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.501205    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb734c3-9bca-4e41-82ad-3bd9575507dc-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.511566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.511533    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:14.543523 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.543387    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:54:14.553602 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.553513    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.553602 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.553550    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.554009 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.553976    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6" pod "94c219d8-565c-4059-9365-0d19a1ec8a3b" (UID: "94c219d8-565c-4059-9365-0d19a1ec8a3b"). InnerVolumeSpecName "kube-api-access-hw7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.554531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.554473    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:14.555942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.555661    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl" pod "8d029cc9-7523-4acb-b82d-cdabd1bad5a6" (UID: "8d029cc9-7523-4acb-b82d-cdabd1bad5a6"). InnerVolumeSpecName "kube-api-access-4mbsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:14.555942 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.555688    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-nw4xn"]
Oct 30 11:54:14.602252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602211    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbsl\" (UniqueName: \"kubernetes.io/projected/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-kube-api-access-4mbsl\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602252 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602257    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602274    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602285    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602293    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602301    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw7z6\" (UniqueName: \"kubernetes.io/projected/94c219d8-565c-4059-9365-0d19a1ec8a3b-kube-api-access-hw7z6\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.602479 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.602314    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d029cc9-7523-4acb-b82d-cdabd1bad5a6-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:14.774670 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.774640    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:54:14.782521 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.782490    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-gb6sx"]
Oct 30 11:54:14.806908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.806818    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:54:14.812421 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.812388    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d9258eb-openshift-default-7b8cd879df-vdvg9"]
Oct 30 11:54:14.912879 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.912841    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" path="/var/lib/kubelet/pods/8d029cc9-7523-4acb-b82d-cdabd1bad5a6/volumes"
Oct 30 11:54:14.913473 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.913451    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" path="/var/lib/kubelet/pods/8fb734c3-9bca-4e41-82ad-3bd9575507dc/volumes"
Oct 30 11:54:14.913963 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:14.913944    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c219d8-565c-4059-9365-0d19a1ec8a3b" path="/var/lib/kubelet/pods/94c219d8-565c-4059-9365-0d19a1ec8a3b/volumes"
Oct 30 11:54:24.978615 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:24.978572    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:54:35.759053 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:35.759020    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:54:35.759645 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:35.759217    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy" containerID="cri-o://1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23" gracePeriod=30
Oct 30 11:54:37.584486 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:37.584439    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:54:41.037826 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.037751    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:54:41.108254 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108208    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108363    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108386    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108420    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108456    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108489    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108521    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:41.108723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108545    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108622    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.108723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.108722    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\") pod \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\" (UID: \"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da\") "
Oct 30 11:54:41.109372 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.109347    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:54:41.109477 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.109375    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.109588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.109558    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:41.109654 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.109598    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:41.109761 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.109737    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:41.111571 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.111533    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:54:41.111692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.111616    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:41.111692 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.111682    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "kube-api-access-x2mxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:54:41.111914 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.111892    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo" pod "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" (UID: "2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:54:41.210780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210745    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.210780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210781    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210794    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210807    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210818    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210830    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mxl\" (UniqueName: \"kubernetes.io/projected/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-kube-api-access-x2mxl\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210843    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.211008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.210854    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:54:41.630754 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.630717    4453 generic.go:356] "Generic (PLEG): container finished" podID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerID="1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23" exitCode=0
Oct 30 11:54:41.630940 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.630778    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"
Oct 30 11:54:41.630940 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.630797    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" event={"ID":"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da","Type":"ContainerDied","Data":"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"}
Oct 30 11:54:41.630940 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.630859    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs" event={"ID":"2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da","Type":"ContainerDied","Data":"287a94859ee97ff5d0ca78f863e5ab22e2fdff46d2fae7bb698888244b27effc"}
Oct 30 11:54:41.630940 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.630889    4453 scope.go:122] "RemoveContainer" containerID="1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"
Oct 30 11:54:41.642588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.641150    4453 scope.go:122] "RemoveContainer" containerID="1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"
Oct 30 11:54:41.642588 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:54:41.641494    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23\": container with ID starting with 1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23 not found: ID does not exist" containerID="1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"
Oct 30 11:54:41.642588 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.641522    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23"} err="failed to get container status \"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23\": rpc error: code = NotFound desc = could not find container \"1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23\": container with ID starting with 1be56cd792fc10edb151e63d4b4ec8f99f3283569a6fa68e990d8556902d6e23 not found: ID does not exist"
Oct 30 11:54:41.682395 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.682360    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:54:41.704148 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:41.704114    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-k9cbs"]
Oct 30 11:54:42.914756 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:54:42.914725    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" path="/var/lib/kubelet/pods/2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da/volumes"
Oct 30 11:55:07.571686 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:55:07.571644    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:55:37.571144 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:55:37.571106    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:56:04.085024 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.084986    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:56:04.085790 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085768    4453 cpu_manager.go:414] "containerMap: removing container" podUID="94c219d8-565c-4059-9365-0d19a1ec8a3b" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085792    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c219d8-565c-4059-9365-0d19a1ec8a3b" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085811    4453 cpu_manager.go:414] "containerMap: removing container" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085818    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085852    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085860    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085881    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy"
Oct 30 11:56:04.085897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085888    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085902    4453 cpu_manager.go:414] "containerMap: removing container" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.085909    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.086081    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb734c3-9bca-4e41-82ad-3bd9575507dc" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.086095    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3fc2b8-93f3-4461-b8f5-4d1ded82b3da" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.086112    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c219d8-565c-4059-9365-0d19a1ec8a3b" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.086124    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="491fc23e-5170-4eba-af16-371e67138787" containerName="istio-proxy"
Oct 30 11:56:04.086241 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.086135    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d029cc9-7523-4acb-b82d-cdabd1bad5a6" containerName="istio-proxy"
Oct 30 11:56:04.090211 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.090189    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.105250 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.105226    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-648839-serving-cert\""
Oct 30 11:56:04.128578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.128548    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:56:04.218863 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.218825    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.219107 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.219088    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.219237 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.219210    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvzj\" (UniqueName: \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.219340 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.219257    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.321176 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.321117    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvzj\" (UniqueName: \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.321176 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.321169    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.321481 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.321235    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.321965 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.321933    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.322222 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:56:04.322102    4453 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-serving-cert: secret "maas-api-e2e-ipp-b-648839-serving-cert" not found
Oct 30 11:56:04.322348 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:56:04.322272    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls podName:e41f682c-b42b-45ee-97bf-1d4bb4b63fab nodeName:}" failed. No retries permitted until 2026-10-30 11:56:04.822251856 +0000 UTC m=+1564.042536760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls") pod "maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" (UID: "e41f682c-b42b-45ee-97bf-1d4bb4b63fab") : secret "maas-api-e2e-ipp-b-648839-serving-cert" not found
Oct 30 11:56:04.322348 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.322292    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.324546 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.324521    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.342301 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.342241    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvzj\" (UniqueName: \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.827825 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.827775    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:04.830235 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:04.830213    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:05.003780 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:05.003741    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:05.330057 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:05.322035    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:56:05.345146 ip-10-0-143-18 kubenswrapper[4453]: W1030 11:56:05.345114    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41f682c_b42b_45ee_97bf_1d4bb4b63fab.slice/crio-c1093a68466731dc0f501b33f67ffda022f78c9226154003979b6d89e09bf2a8 WatchSource:0}: Error finding container c1093a68466731dc0f501b33f67ffda022f78c9226154003979b6d89e09bf2a8: Status 404 returned error can't find the container with id c1093a68466731dc0f501b33f67ffda022f78c9226154003979b6d89e09bf2a8
Oct 30 11:56:06.228403 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:06.228364    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" event={"ID":"e41f682c-b42b-45ee-97bf-1d4bb4b63fab","Type":"ContainerStarted","Data":"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"}
Oct 30 11:56:06.228585 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:06.228430    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" event={"ID":"e41f682c-b42b-45ee-97bf-1d4bb4b63fab","Type":"ContainerStarted","Data":"c1093a68466731dc0f501b33f67ffda022f78c9226154003979b6d89e09bf2a8"}
Oct 30 11:56:06.228585 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:06.228463    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:06.270577 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:06.270534    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" podStartSLOduration=1.955688168 podStartE2EDuration="2.27051881s" totalImagesPullingTime="314.830642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:56:06.26700978 +0000 UTC m=+1565.487294822" watchObservedRunningTime="2026-10-30 11:56:06.27051881 +0000 UTC m=+1565.490803719"
Oct 30 11:56:07.569750 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:56:07.569718    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:56:12.240139 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:12.240108    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:56:24.979183 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:56:24.979105    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 11:56:37.570777 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:56:37.570727    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:57:07.571515 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:57:07.571472    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:57:31.399850 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:57:31.399811    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 11:57:31.617385 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:57:31.617355    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 11:57:37.571051 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:57:37.571007    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:58:07.572361 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:58:07.572306    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:58:11.975674 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:11.975641    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:58:11.976190 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:11.975891    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerName="maas-api" containerID="cri-o://6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951" gracePeriod=30
Oct 30 11:58:12.256641 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.256614    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:58:12.345271 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.345226    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\" (UniqueName: \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\") pod \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") "
Oct 30 11:58:12.345483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.345374    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") pod \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") "
Oct 30 11:58:12.345483 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.345450    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\") pod \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") "
Oct 30 11:58:12.345610 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.345554    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\") pod \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\" (UID: \"e41f682c-b42b-45ee-97bf-1d4bb4b63fab\") "
Oct 30 11:58:12.346036 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.346008    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca" pod "e41f682c-b42b-45ee-97bf-1d4bb4b63fab" (UID: "e41f682c-b42b-45ee-97bf-1d4bb4b63fab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:12.346169 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.346112    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:12.347800 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.347776    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj" pod "e41f682c-b42b-45ee-97bf-1d4bb4b63fab" (UID: "e41f682c-b42b-45ee-97bf-1d4bb4b63fab"). InnerVolumeSpecName "kube-api-access-htvzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:12.347897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.347795    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs" pod "e41f682c-b42b-45ee-97bf-1d4bb4b63fab" (UID: "e41f682c-b42b-45ee-97bf-1d4bb4b63fab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:12.347897 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.347822    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls" pod "e41f682c-b42b-45ee-97bf-1d4bb4b63fab" (UID: "e41f682c-b42b-45ee-97bf-1d4bb4b63fab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:12.447908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.447841    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htvzj\" (UniqueName: \"kubernetes.io/projected/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-kube-api-access-htvzj\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:12.447908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.447883    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:12.447908 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:12.447898    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e41f682c-b42b-45ee-97bf-1d4bb4b63fab-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:13.045615 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.045583    4453 generic.go:356] "Generic (PLEG): container finished" podID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerID="6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951" exitCode=0
Oct 30 11:58:13.046035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.045623    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"
Oct 30 11:58:13.046035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.045665    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" event={"ID":"e41f682c-b42b-45ee-97bf-1d4bb4b63fab","Type":"ContainerDied","Data":"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"}
Oct 30 11:58:13.046035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.045719    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" event={"ID":"e41f682c-b42b-45ee-97bf-1d4bb4b63fab","Type":"ContainerDied","Data":"c1093a68466731dc0f501b33f67ffda022f78c9226154003979b6d89e09bf2a8"}
Oct 30 11:58:13.046035 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.045743    4453 scope.go:122] "RemoveContainer" containerID="6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"
Oct 30 11:58:13.055228 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.055213    4453 scope.go:122] "RemoveContainer" containerID="6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"
Oct 30 11:58:13.055511 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:58:13.055493    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951\": container with ID starting with 6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951 not found: ID does not exist" containerID="6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"
Oct 30 11:58:13.055565 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.055518    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951"} err="failed to get container status \"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951\": rpc error: code = NotFound desc = could not find container \"6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951\": container with ID starting with 6137e644e42acbca24b29a2ea64a86233318ae988e22a40d5321b96228a43951 not found: ID does not exist"
Oct 30 11:58:13.120833 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.120803    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:58:13.144566 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:13.144540    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4"]
Oct 30 11:58:14.913135 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:14.913099    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" path="/var/lib/kubelet/pods/e41f682c-b42b-45ee-97bf-1d4bb4b63fab/volumes"
Oct 30 11:58:15.245723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:15.245632    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648839-6978b6fcf6-tsbc4" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.88:8443/health\": context deadline exceeded"
Oct 30 11:58:18.967747 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:18.967707    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:58:18.968166 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:18.967970    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api" containerID="cri-o://a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9" gracePeriod=30
Oct 30 11:58:19.459437 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.459411    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:58:19.642109 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.642077    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\") pod \"a7af338e-a969-49e5-8468-25754331e968\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") "
Oct 30 11:58:19.642392 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.642180    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\" (UniqueName: \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\") pod \"a7af338e-a969-49e5-8468-25754331e968\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") "
Oct 30 11:58:19.642392 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.642377    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\") pod \"a7af338e-a969-49e5-8468-25754331e968\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") "
Oct 30 11:58:19.642722 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.642470    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\") pod \"a7af338e-a969-49e5-8468-25754331e968\" (UID: \"a7af338e-a969-49e5-8468-25754331e968\") "
Oct 30 11:58:19.644953 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.642987    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca" pod "a7af338e-a969-49e5-8468-25754331e968" (UID: "a7af338e-a969-49e5-8468-25754331e968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:58:19.645398 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.645370    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs" pod "a7af338e-a969-49e5-8468-25754331e968" (UID: "a7af338e-a969-49e5-8468-25754331e968"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:19.645687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.645654    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r" pod "a7af338e-a969-49e5-8468-25754331e968" (UID: "a7af338e-a969-49e5-8468-25754331e968"). InnerVolumeSpecName "kube-api-access-bnv8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:58:19.646437 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.646416    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls" pod "a7af338e-a969-49e5-8468-25754331e968" (UID: "a7af338e-a969-49e5-8468-25754331e968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 11:58:19.760351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.747539    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv8r\" (UniqueName: \"kubernetes.io/projected/a7af338e-a969-49e5-8468-25754331e968-kube-api-access-bnv8r\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:19.760351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.747580    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7af338e-a969-49e5-8468-25754331e968-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:19.760351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.747596    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:19.760351 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:19.747609    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7af338e-a969-49e5-8468-25754331e968-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:58:20.094810 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.094729    4453 generic.go:356] "Generic (PLEG): container finished" podID="a7af338e-a969-49e5-8468-25754331e968" containerID="a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9" exitCode=0
Oct 30 11:58:20.094810 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.094796    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"
Oct 30 11:58:20.095312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.094805    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerDied","Data":"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"}
Oct 30 11:58:20.095312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.094859    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb" event={"ID":"a7af338e-a969-49e5-8468-25754331e968","Type":"ContainerDied","Data":"5238f1c0eb21bca48ae259fa9f326a34aed584d9146a4a7dcec3bdfa76d3d723"}
Oct 30 11:58:20.095312 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.094883    4453 scope.go:122] "RemoveContainer" containerID="a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"
Oct 30 11:58:20.106369 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.106343    4453 scope.go:122] "RemoveContainer" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:58:20.118366 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.118347    4453 scope.go:122] "RemoveContainer" containerID="a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"
Oct 30 11:58:20.118681 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:58:20.118661    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9\": container with ID starting with a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9 not found: ID does not exist" containerID="a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"
Oct 30 11:58:20.118745 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.118691    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9"} err="failed to get container status \"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9\": rpc error: code = NotFound desc = could not find container \"a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9\": container with ID starting with a592649ed431a6a3d2ea8c544572c85c15be9f27efd224c5d13eb0e4cba7bbe9 not found: ID does not exist"
Oct 30 11:58:20.118745 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.118710    4453 scope.go:122] "RemoveContainer" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:58:20.119006 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:58:20.118988    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e\": container with ID starting with 3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e not found: ID does not exist" containerID="3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"
Oct 30 11:58:20.119058 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.119016    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e"} err="failed to get container status \"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e\": rpc error: code = NotFound desc = could not find container \"3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e\": container with ID starting with 3c6a9a0dd141fd8889552c729dba36292804f89665c0a9bcc9b120ab976bb03e not found: ID does not exist"
Oct 30 11:58:20.156233 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.156203    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:58:20.183386 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.183312    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-caa7a0-65dc54f7fb-4mgxb"]
Oct 30 11:58:20.913578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:58:20.913543    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7af338e-a969-49e5-8468-25754331e968" path="/var/lib/kubelet/pods/a7af338e-a969-49e5-8468-25754331e968/volumes"
Oct 30 11:58:37.570496 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:58:37.570460    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:59:05.111212 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:05.111181    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:59:05.111803 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:05.111398    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy" containerID="cri-o://c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51" gracePeriod=30
Oct 30 11:59:05.120514 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:05.120487    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:59:05.120674 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:05.120654    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy" containerID="cri-o://a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10" gracePeriod=30
Oct 30 11:59:07.573559 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:07.573516    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:59:10.430572 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.430539    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:59:10.478687 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.478659    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:59:10.486723 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.486689    4453 generic.go:356] "Generic (PLEG): container finished" podID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerID="a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10" exitCode=0
Oct 30 11:59:10.486860 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.486733    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" event={"ID":"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4","Type":"ContainerDied","Data":"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"}
Oct 30 11:59:10.486860 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.486762    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"
Oct 30 11:59:10.486860 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.486822    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6" event={"ID":"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4","Type":"ContainerDied","Data":"76a63d424a6683bbff43f37d818a09a939b36c7d39787376cd11b2bf19b81911"}
Oct 30 11:59:10.486860 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.486852    4453 scope.go:122] "RemoveContainer" containerID="a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"
Oct 30 11:59:10.489086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.489063    4453 generic.go:356] "Generic (PLEG): container finished" podID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerID="c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51" exitCode=0
Oct 30 11:59:10.489086 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.489091    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" event={"ID":"492ee220-3c69-4b52-ab7d-ec1371ee1c8c","Type":"ContainerDied","Data":"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"}
Oct 30 11:59:10.489400 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.489127    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f" event={"ID":"492ee220-3c69-4b52-ab7d-ec1371ee1c8c","Type":"ContainerDied","Data":"b8cb73e58159e83cfada04b58dbc4224b2a9c00602e08b40cc97cf7629147739"}
Oct 30 11:59:10.489400 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.489126    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"
Oct 30 11:59:10.500304 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.500284    4453 scope.go:122] "RemoveContainer" containerID="a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"
Oct 30 11:59:10.500706 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:10.500680    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10\": container with ID starting with a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10 not found: ID does not exist" containerID="a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"
Oct 30 11:59:10.500789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.500720    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10"} err="failed to get container status \"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10\": rpc error: code = NotFound desc = could not find container \"a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10\": container with ID starting with a9c1e5e3e5164589ea18e0f72aae2c7f2a92e0dfb9c5f6c001db47382bfdda10 not found: ID does not exist"
Oct 30 11:59:10.500789 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.500743    4453 scope.go:122] "RemoveContainer" containerID="c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"
Oct 30 11:59:10.510614 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.510593    4453 scope.go:122] "RemoveContainer" containerID="c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"
Oct 30 11:59:10.510999 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:10.510976    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51\": container with ID starting with c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51 not found: ID does not exist" containerID="c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"
Oct 30 11:59:10.511087 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.511011    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51"} err="failed to get container status \"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51\": rpc error: code = NotFound desc = could not find container \"c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51\": container with ID starting with c495b31c7d889af4673a393565b02f5e4da600789f38a9485df23e5254a2ef51 not found: ID does not exist"
Oct 30 11:59:10.539548 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539515    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.539741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539574    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.539741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539608    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.539741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539698    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.539741 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539733    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539772    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539809    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539916    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539953    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539981    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540020    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540052    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540103 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540098    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540148    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540189    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539947    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.539980    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540200    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540292    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540383    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\") pod \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\" (UID: \"492ee220-3c69-4b52-ab7d-ec1371ee1c8c\") "
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540462    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.540555 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540497    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\") pod \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\" (UID: \"b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4\") "
Oct 30 11:59:10.541011 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540675    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.541011 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.540710    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541191    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541217    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541232    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541246    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541258    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541270    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.541533 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541449    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.541990 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541581    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.541990 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.541783    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.542142 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.542118    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:10.544130 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544104    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544627    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544631    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544762    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "kube-api-access-46hk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544808    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544818    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo" pod "492ee220-3c69-4b52-ab7d-ec1371ee1c8c" (UID: "492ee220-3c69-4b52-ab7d-ec1371ee1c8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:10.545401 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.544860    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:10.545857 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.545838    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8" pod "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" (UID: "b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4"). InnerVolumeSpecName "kube-api-access-7lnw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:10.642158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642119    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnw8\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-kube-api-access-7lnw8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642148    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642158    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642158 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642166    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642174    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642182    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642190    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642198    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642206    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642216    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hk8\" (UniqueName: \"kubernetes.io/projected/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-kube-api-access-46hk8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642224    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.642451 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.642232    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/492ee220-3c69-4b52-ab7d-ec1371ee1c8c-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:10.834143 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.834110    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:59:10.843342 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.843300    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-7zc4f"]
Oct 30 11:59:10.866184 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.866152    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:59:10.870921 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.870895    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-skqp6"]
Oct 30 11:59:10.911916 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.911883    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" path="/var/lib/kubelet/pods/492ee220-3c69-4b52-ab7d-ec1371ee1c8c/volumes"
Oct 30 11:59:10.912433 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:10.912416    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" path="/var/lib/kubelet/pods/b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4/volumes"
Oct 30 11:59:37.166447 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.166415    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 11:59:37.167020 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167004    4453 cpu_manager.go:414] "containerMap: removing container" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167022    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167040    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167045    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167059    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167065    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167071    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167075    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167085    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerName="maas-api"
Oct 30 11:59:37.167099 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167089    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerName="maas-api"
Oct 30 11:59:37.167578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167198    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="492ee220-3c69-4b52-ab7d-ec1371ee1c8c" containerName="istio-proxy"
Oct 30 11:59:37.167578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167206    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fcdbf9-9d8d-4be0-8e82-172e6d266ef4" containerName="istio-proxy"
Oct 30 11:59:37.167578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167213    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167218    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 11:59:37.167578 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.167226    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41f682c-b42b-45ee-97bf-1d4bb4b63fab" containerName="maas-api"
Oct 30 11:59:37.170481 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.170461    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.183046 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.183022    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e25f4a-openshift-default-dockercfg-fp4tl\""
Oct 30 11:59:37.197525 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.197499    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 11:59:37.297513 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297478    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.297691 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297531    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz962\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.297765 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297690    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.297838 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297815    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.297934 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297916    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.297994 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.297976    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.298044 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.298027    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.298089 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.298074    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.298136 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.298100    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399020 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.398983    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399106    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399147    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399171    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399222 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399211    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399468 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399250    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399468 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399273    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399468 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399358    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399468 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399390    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz962\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399717 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399689    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399885 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399822    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.399998 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.399975    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.400065 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.400046    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.400126 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.400071    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.401654 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.401628    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.401764 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.401753    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.420714 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.420659    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.420829 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.420811    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz962\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.487397 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.487377    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:37.585431 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:37.580905    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 11:59:37.801326 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.801291    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 11:59:37.807777 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.807743    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:59:37.807868 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.807828    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:59:37.807868 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:37.807856    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 11:59:38.656458 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:38.656423    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" event={"ID":"17b18771-e8ec-4556-adf9-10d2ac283c48","Type":"ContainerStarted","Data":"9d253565f6b009232db40ae81f8b262f1bf31df3394eb88dc12b9eb4a79eaefc"}
Oct 30 11:59:38.656906 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:38.656477    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" event={"ID":"17b18771-e8ec-4556-adf9-10d2ac283c48","Type":"ContainerStarted","Data":"772cde8607b24422a18dd5b9a19c19ef6f166ef6f041c06d883e712047cef656"}
Oct 30 11:59:38.719905 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:38.719849    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podStartSLOduration=1.7198339470000001 podStartE2EDuration="1.719833947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 11:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 11:59:38.707986989 +0000 UTC m=+1777.928271900" watchObservedRunningTime="2026-10-30 11:59:38.719833947 +0000 UTC m=+1777.940118856"
Oct 30 11:59:39.512354 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:39.488628    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:39.513010 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:39.512649    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:59:39.513010 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:39.512723    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 11:59:40.488014 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:40.487982    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:59:40.488468 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:40.488053    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 11:59:41.487869 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:41.487833    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:59:41.488052 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:41.487895    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 11:59:42.488208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:42.488170    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:59:42.488728 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:42.488235    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 11:59:43.511506 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:43.511472    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 11:59:43.512078 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:43.512043    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 11:59:44.492905 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:44.492861    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:44.493256 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:44.493236    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:44.493988 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:44.493971    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 11:59:49.774982 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:49.774946    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:59:49.775427 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:49.775203    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy" containerID="cri-o://9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c" gracePeriod=30
Oct 30 11:59:55.036801 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.036776    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:59:55.123138 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123094    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123146    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123224    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123255    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123303    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123371    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123411    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123447    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.123531 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123504    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\") pod \"6e26873a-a86e-4153-a051-1f658c8650ad\" (UID: \"6e26873a-a86e-4153-a051-1f658c8650ad\") "
Oct 30 11:59:55.124008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123545    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:55.124008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123629    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:55.124008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123881    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:55.124008 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.123987    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 11:59:55.124234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.124045    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.124234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.124065    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.124234 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.124079    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.124415 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.124299    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:55.126644 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.126610    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "kube-api-access-2ts5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:55.126865 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.126835    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 11:59:55.126987 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.126863    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 11:59:55.126987 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.126915    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token" pod "6e26873a-a86e-4153-a051-1f658c8650ad" (UID: "6e26873a-a86e-4153-a051-1f658c8650ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 11:59:55.225303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225274    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.225303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225305    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e26873a-a86e-4153-a051-1f658c8650ad-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.225303 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225318    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e26873a-a86e-4153-a051-1f658c8650ad-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.225563 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225356    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.225563 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225369    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e26873a-a86e-4153-a051-1f658c8650ad-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.225563 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.225383    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ts5f\" (UniqueName: \"kubernetes.io/projected/6e26873a-a86e-4153-a051-1f658c8650ad-kube-api-access-2ts5f\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 11:59:55.855208 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.855166    4453 generic.go:356] "Generic (PLEG): container finished" podID="6e26873a-a86e-4153-a051-1f658c8650ad" containerID="9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c" exitCode=0
Oct 30 11:59:55.855510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.855227    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"
Oct 30 11:59:55.855510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.855278    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerDied","Data":"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"}
Oct 30 11:59:55.855510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.855403    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764" event={"ID":"6e26873a-a86e-4153-a051-1f658c8650ad","Type":"ContainerDied","Data":"7cd56a320d12842484e3c6cd94dd0aef4b821896e77dc708a7e8ef552aea8888"}
Oct 30 11:59:55.855510 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.855442    4453 scope.go:122] "RemoveContainer" containerID="9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"
Oct 30 11:59:55.872411 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.872379    4453 scope.go:122] "RemoveContainer" containerID="dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"
Oct 30 11:59:55.881504 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.881475    4453 scope.go:122] "RemoveContainer" containerID="9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"
Oct 30 11:59:55.881824 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:55.881790    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c\": container with ID starting with 9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c not found: ID does not exist" containerID="9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"
Oct 30 11:59:55.881898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.881833    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c"} err="failed to get container status \"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c\": rpc error: code = NotFound desc = could not find container \"9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c\": container with ID starting with 9ad428c379a211d7be501919432692849f533cd2331942794d8ce43b6f45822c not found: ID does not exist"
Oct 30 11:59:55.881898 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.881854    4453 scope.go:122] "RemoveContainer" containerID="dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"
Oct 30 11:59:55.882123 ip-10-0-143-18 kubenswrapper[4453]: E1030 11:59:55.882106    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9\": container with ID starting with dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9 not found: ID does not exist" containerID="dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"
Oct 30 11:59:55.882169 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.882130    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9"} err="failed to get container status \"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9\": rpc error: code = NotFound desc = could not find container \"dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9\": container with ID starting with dac58026bdceabd7eed7133a915e94b81a061db6c201bcc891f15bfbe7302de9 not found: ID does not exist"
Oct 30 11:59:55.922805 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.922772    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:59:55.931416 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:55.931383    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3ccdab-openshift-default-6865b8777c-9v764"]
Oct 30 11:59:56.911064 ip-10-0-143-18 kubenswrapper[4453]: I1030 11:59:56.911030    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" path="/var/lib/kubelet/pods/6e26873a-a86e-4153-a051-1f658c8650ad/volumes"
Oct 30 12:00:07.287809 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.287778    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:00:07.288591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288569    4453 cpu_manager.go:414] "containerMap: removing container" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.288591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288590    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.288777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288602    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 12:00:07.288777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288610    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 12:00:07.288777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288627    4453 cpu_manager.go:414] "containerMap: removing container" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.288777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288634    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.288973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288817    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.288973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288829    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7af338e-a969-49e5-8468-25754331e968" containerName="maas-api"
Oct 30 12:00:07.288973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.288840    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e26873a-a86e-4153-a051-1f658c8650ad" containerName="istio-proxy"
Oct 30 12:00:07.293453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.293431    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.321859 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.321825    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:00:07.350467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350435    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.350621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350490    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.350693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350614    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.350693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350663    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.351083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350694    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.351083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350767    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.351083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350794    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.351083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350842    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjp2t\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.351083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.350887    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452370    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452416    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452442    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452515    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452542    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452592    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjp2t\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452643    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452773    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452947 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452825    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452947 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452826    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.452947 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.452903    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.453164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.453141    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.453250 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.453229    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.453411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.453392    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.454903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.454884    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.455256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.455241    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.474095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.474074    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.474254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.474235    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjp2t\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.568898 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:00:07.568826    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:00:07.606679 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.606650    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:07.924515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.924484    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:00:07.926151 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:00:07.926125    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbe00a0_6044_4618_9fe6_de23e959b0af.slice/crio-b2ab7cf108e29d9c52851577e6809d0fa69896428d7460c74875c51673f1ec21 WatchSource:0}: Error finding container b2ab7cf108e29d9c52851577e6809d0fa69896428d7460c74875c51673f1ec21: Status 404 returned error can't find the container with id b2ab7cf108e29d9c52851577e6809d0fa69896428d7460c74875c51673f1ec21
Oct 30 12:00:07.928616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.928581    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:07.928680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.928667    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:07.928716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.928697    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:07.945509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:07.945484    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" event={"ID":"6dbe00a0-6044-4618-9fe6-de23e959b0af","Type":"ContainerStarted","Data":"b2ab7cf108e29d9c52851577e6809d0fa69896428d7460c74875c51673f1ec21"}
Oct 30 12:00:08.952558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:08.952523    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" event={"ID":"6dbe00a0-6044-4618-9fe6-de23e959b0af","Type":"ContainerStarted","Data":"9a8e60bb2fdfcadc351d1dc064bc9b275dffca47955b7d89922c59e75266a173"}
Oct 30 12:00:09.014731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:09.014663    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podStartSLOduration=2.014640195 podStartE2EDuration="2.014640195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:09.009040364 +0000 UTC m=+1808.229325277" watchObservedRunningTime="2026-10-30 12:00:09.014640195 +0000 UTC m=+1808.234925107"
Oct 30 12:00:09.614042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:09.613961    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:09.616112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:09.616068    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:00:09.616251 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:09.616130    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:00:10.611355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:10.610749    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:00:10.611355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:10.610815    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:00:11.608757 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:11.608712    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:00:11.609041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:11.609013    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:00:12.613734 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:12.613697    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:00:12.614241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:12.614201    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:00:13.613493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:13.613460    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 12:00:13.613746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:13.613712    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 12:00:14.611981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:14.611951    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:14.612410 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:14.612396    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:14.612992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:14.612977    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:00:22.314893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.314861    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:00:22.319066 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.319048    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344067 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344040    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344195 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344171    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344212    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344256    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344284    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344523    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344571    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344673    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.344760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.344742    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xhf\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.346261 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.346233    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:00:22.446439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446403    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446448    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446511    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446529    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446649    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446741    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xhf\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446828    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.446907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446828    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446954    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.446992    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.447256    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.447389    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.447454    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.447485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.447463    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.448912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.448885    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.449222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.449201    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.470975 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.470937    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.471726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.471707    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xhf\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\") pod \"e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") " pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.632824 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.632790    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:22.950021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.949993    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:00:22.950984 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:00:22.950955    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb42af7_0b56_4a24_9ac6_a86c756b25cc.slice/crio-88651f49e577907cb47a3d961d38bad45fa4b538315f24ce57458453df28ab07 WatchSource:0}: Error finding container 88651f49e577907cb47a3d961d38bad45fa4b538315f24ce57458453df28ab07: Status 404 returned error can't find the container with id 88651f49e577907cb47a3d961d38bad45fa4b538315f24ce57458453df28ab07
Oct 30 12:00:22.953323 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.953289    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:22.953458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.953408    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:22.953521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:22.953465    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:00:23.041853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:23.041803    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" event={"ID":"1cb42af7-0b56-4a24-9ac6-a86c756b25cc","Type":"ContainerStarted","Data":"3aff1f58abd22e4b8ba628c217e9359911b923dc6ed9cac2ce45f0d904edcde2"}
Oct 30 12:00:23.042043 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:23.041873    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" event={"ID":"1cb42af7-0b56-4a24-9ac6-a86c756b25cc","Type":"ContainerStarted","Data":"88651f49e577907cb47a3d961d38bad45fa4b538315f24ce57458453df28ab07"}
Oct 30 12:00:23.086272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:23.086206    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podStartSLOduration=1.0861868 podStartE2EDuration="1.0861868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:00:23.082067843 +0000 UTC m=+1822.302352756" watchObservedRunningTime="2026-10-30 12:00:23.0861868 +0000 UTC m=+1822.306471711"
Oct 30 12:00:23.633052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:23.633016    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:24.635345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:24.635303    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 12:00:24.635748 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:24.635371    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 12:00:25.633511 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:25.633474    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 12:00:25.633698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:25.633528    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 12:00:26.633228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:26.633192    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 12:00:26.633639 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:26.633254    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 12:00:27.633279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:27.633240    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 12:00:27.633699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:27.633303    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 12:00:28.633230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:28.633194    4453 patch_prober.go:29] interesting pod/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 12:00:28.633442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:28.633254    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 12:00:29.638811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:29.638773    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:29.639265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:29.639245    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:29.639848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:00:29.639827    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:00:37.574003 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:00:37.573958    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:01:05.492934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.492899    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:01:05.497282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.497256    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.525294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.525268    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:01:05.575748 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.575717    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggp9\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.575932 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.575755    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.575932 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.575891    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576067 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.575936    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576067 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.576040    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.576094    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.576189    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.576231    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.576284 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.576248    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677360    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677583 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677407    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677583 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677464    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677583 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677511    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677616    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677682    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677708    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677765    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggp9\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.677825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677796    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.678065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677853    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.678065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.677930    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.678065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.678005    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.678065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.678035    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.678529 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.678499    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.679802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.679784    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.680148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.680130    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.698286 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.698263    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.698442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.698364    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggp9\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:05.812514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:05.812451    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:06.127360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.127303    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:01:06.127741 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:01:06.127708    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43388545_8b63_4978_a970_19a398a270b6.slice/crio-5cd04074f71f486eda5db569231ec9421d686d9985206f6d667ac5410ed793da WatchSource:0}: Error finding container 5cd04074f71f486eda5db569231ec9421d686d9985206f6d667ac5410ed793da: Status 404 returned error can't find the container with id 5cd04074f71f486eda5db569231ec9421d686d9985206f6d667ac5410ed793da
Oct 30 12:01:06.130309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.130277    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:01:06.130406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.130373    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:01:06.130448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.130403    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:01:06.314875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.314840    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" event={"ID":"43388545-8b63-4978-a970-19a398a270b6","Type":"ContainerStarted","Data":"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"}
Oct 30 12:01:06.315055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.314903    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" event={"ID":"43388545-8b63-4978-a970-19a398a270b6","Type":"ContainerStarted","Data":"5cd04074f71f486eda5db569231ec9421d686d9985206f6d667ac5410ed793da"}
Oct 30 12:01:06.357929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.357874    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podStartSLOduration=1.357858969 podStartE2EDuration="1.357858969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:01:06.354532787 +0000 UTC m=+1865.574817698" watchObservedRunningTime="2026-10-30 12:01:06.357858969 +0000 UTC m=+1865.578143879"
Oct 30 12:01:06.812982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:06.812948    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:07.570816 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:01:07.570782    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:01:07.815272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:07.815237    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:07.815696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:07.815293    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:08.813821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:08.813776    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:08.814010 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:08.813844    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:09.814030 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:09.813662    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:09.814030 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:09.813721    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:10.813783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:10.813739    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:10.813985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:10.813821    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:11.815354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:11.814739    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:11.815354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:11.814807    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:12.813422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:12.813388    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 12:01:12.813654 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:12.813443    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 12:01:14.606845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:14.606807    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:14.607588 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:14.607552    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:14.609199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:14.609170    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:01:37.593786 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:01:37.593746    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:01:56.842630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:56.842595    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 12:01:56.843095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:01:56.842851    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy" containerID="cri-o://f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e" gracePeriod=30
Oct 30 12:02:02.114308 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.114281    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 12:02:02.155862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.155785    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.155862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.155829    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.155862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.155848    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.155880    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.155943    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156085    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156114    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156124    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:02.156404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156369    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.156465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156406    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:02:02.156624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156604    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:02.156707 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156616    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:02.156707 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156699    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\") pod \"87e2f508-5903-41ac-babf-991306dcf37d\" (UID: \"87e2f508-5903-41ac-babf-991306dcf37d\") "
Oct 30 12:02:02.157009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.156975    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:02.157475 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.157451    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e2f508-5903-41ac-babf-991306dcf37d-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.157565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.157479    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.157565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.157493    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.157565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.157506    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.157565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.157523    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.158344 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.158311    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:02:02.158437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.158353    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:02:02.158437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.158321    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:02.158437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.158410    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7" pod "87e2f508-5903-41ac-babf-991306dcf37d" (UID: "87e2f508-5903-41ac-babf-991306dcf37d"). InnerVolumeSpecName "kube-api-access-5hwl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:02:02.258803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.258768    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e2f508-5903-41ac-babf-991306dcf37d-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.258803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.258797    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.258803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.258806    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e2f508-5903-41ac-babf-991306dcf37d-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.259045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.258815    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwl7\" (UniqueName: \"kubernetes.io/projected/87e2f508-5903-41ac-babf-991306dcf37d-kube-api-access-5hwl7\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:02:02.727533 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.727493    4453 generic.go:356] "Generic (PLEG): container finished" podID="87e2f508-5903-41ac-babf-991306dcf37d" containerID="f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e" exitCode=0
Oct 30 12:02:02.727744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.727539    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"
Oct 30 12:02:02.727744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.727586    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" event={"ID":"87e2f508-5903-41ac-babf-991306dcf37d","Type":"ContainerDied","Data":"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"}
Oct 30 12:02:02.727744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.727639    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx" event={"ID":"87e2f508-5903-41ac-babf-991306dcf37d","Type":"ContainerDied","Data":"6cdefbbb279cc02c804e30d3eb306640f4070d397080f4887e931e7c7e0e8836"}
Oct 30 12:02:02.727744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.727661    4453 scope.go:122] "RemoveContainer" containerID="f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"
Oct 30 12:02:02.737905 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.737888    4453 scope.go:122] "RemoveContainer" containerID="f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"
Oct 30 12:02:02.738242 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:02:02.738206    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e\": container with ID starting with f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e not found: ID does not exist" containerID="f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"
Oct 30 12:02:02.738379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.738251    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e"} err="failed to get container status \"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e\": rpc error: code = NotFound desc = could not find container \"f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e\": container with ID starting with f8f2b962174b07e2f03ff6481278ff68aef39706f590ee7afad5edc1c098981e not found: ID does not exist"
Oct 30 12:02:02.783145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.783112    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 12:02:02.792710 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.792678    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-ncwtx"]
Oct 30 12:02:02.919434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:02.919399    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e2f508-5903-41ac-babf-991306dcf37d" path="/var/lib/kubelet/pods/87e2f508-5903-41ac-babf-991306dcf37d/volumes"
Oct 30 12:02:05.678849 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.678811    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:02:05.679419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.679403    4453 cpu_manager.go:414] "containerMap: removing container" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy"
Oct 30 12:02:05.679419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.679419    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy"
Oct 30 12:02:05.679544 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.679533    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e2f508-5903-41ac-babf-991306dcf37d" containerName="istio-proxy"
Oct 30 12:02:05.684634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.684606    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.728982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.728951    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:02:05.831754 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.831715    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.831951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.831780    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.831951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.831822    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m4p\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.831958    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832156 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.832117    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.832190    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.832265    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.832412    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.832514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.832470    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.933949    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934083    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934135    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934221    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934256    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934397    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934400    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934440    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934500    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m4p\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934566    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.934850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.934657    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.935350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.935305    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.935599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.935575    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.935692 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.935581    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.937950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.937904    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.938747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.938722    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.956549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.956487    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:05.956656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:05.956581    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m4p\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\") pod \"e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") " pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:06.024345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.024291    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:06.355843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.355819    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:02:06.356895 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:02:06.356863    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2357682_dde3_428d_87d2_3f9cc8055270.slice/crio-9d765b038330bb18a7eb2413a38ad4c9969a224a6a80d48a48fd1e3c99109736 WatchSource:0}: Error finding container 9d765b038330bb18a7eb2413a38ad4c9969a224a6a80d48a48fd1e3c99109736: Status 404 returned error can't find the container with id 9d765b038330bb18a7eb2413a38ad4c9969a224a6a80d48a48fd1e3c99109736
Oct 30 12:02:06.359392 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.359358    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:02:06.359492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.359424    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:02:06.359492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.359455    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:02:06.846624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.846587    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" event={"ID":"a2357682-dde3-428d-87d2-3f9cc8055270","Type":"ContainerStarted","Data":"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"}
Oct 30 12:02:06.847010 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.846671    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" event={"ID":"a2357682-dde3-428d-87d2-3f9cc8055270","Type":"ContainerStarted","Data":"9d765b038330bb18a7eb2413a38ad4c9969a224a6a80d48a48fd1e3c99109736"}
Oct 30 12:02:06.851088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.851064    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 12:02:06.888592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:06.888538    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podStartSLOduration=1.888524375 podStartE2EDuration="1.888524375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:02:06.88527888 +0000 UTC m=+1926.105563790" watchObservedRunningTime="2026-10-30 12:02:06.888524375 +0000 UTC m=+1926.108809284"
Oct 30 12:02:07.025410 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:07.025374    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:07.027233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:07.027201    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:07.027407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:07.027258    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:07.066965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:07.066940    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 12:02:07.571098 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:02:07.571067    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:02:08.024723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:08.024678    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:08.025156 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:08.024736    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:09.025556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:09.025517    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:09.026032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:09.025585    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:10.025531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:10.025443    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:10.025531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:10.025498    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:11.025224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:11.025186    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:11.025423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:11.025266    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:12.024788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:12.024751    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:12.025188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:12.024817    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:13.025578 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:13.025541    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 12:02:13.026132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:13.025641    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 12:02:14.030185 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:14.030108    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:14.030584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:14.030542    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:14.031126 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:02:14.031107    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:02:37.571667 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:02:37.571620    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:03:07.580676 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:03:07.580631    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:03:28.587831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:28.587793    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 12:03:28.588319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:28.588045    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy" containerID="cri-o://b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038" gracePeriod=30
Oct 30 12:03:33.873565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.873505    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 12:03:33.941218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941192    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941226    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941271    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941318    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941548 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941391    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941548 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941429    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941548 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941500    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:33.941706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941622    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:33.941770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941710    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:03:33.941837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941788    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:33.941913 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.941855    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:33.942225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.942203    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f99652-ea34-4cb3-ad33-55471d784e9b-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:33.942611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.942595    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:33.942729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.942713    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:33.942799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.942731    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:33.944212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.944185    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:33.944295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.944225    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "kube-api-access-4dzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:33.944516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:33.944492    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:03:34.043371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.043313    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.043401    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\") pod \"27f99652-ea34-4cb3-ad33-55471d784e9b\" (UID: \"27f99652-ea34-4cb3-ad33-55471d784e9b\") "
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.044000    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.044194    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.044217    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.044232    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzdp\" (UniqueName: \"kubernetes.io/projected/27f99652-ea34-4cb3-ad33-55471d784e9b-kube-api-access-4dzdp\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:34.044944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.044246    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:34.048160 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.048106    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo" pod "27f99652-ea34-4cb3-ad33-55471d784e9b" (UID: "27f99652-ea34-4cb3-ad33-55471d784e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:03:34.145992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.145924    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f99652-ea34-4cb3-ad33-55471d784e9b-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:03:34.494591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.494504    4453 generic.go:356] "Generic (PLEG): container finished" podID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerID="b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038" exitCode=0
Oct 30 12:03:34.494591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.494551    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"
Oct 30 12:03:34.494591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.494565    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" event={"ID":"27f99652-ea34-4cb3-ad33-55471d784e9b","Type":"ContainerDied","Data":"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"}
Oct 30 12:03:34.494856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.494611    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr" event={"ID":"27f99652-ea34-4cb3-ad33-55471d784e9b","Type":"ContainerDied","Data":"eb1ebb8b536803c8f3cfe7c0515223f7835047a5a399e0ad0735ff9651cbb4fd"}
Oct 30 12:03:34.494856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.494634    4453 scope.go:122] "RemoveContainer" containerID="b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"
Oct 30 12:03:34.504946 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.504928    4453 scope.go:122] "RemoveContainer" containerID="b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"
Oct 30 12:03:34.505249 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:03:34.505229    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038\": container with ID starting with b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038 not found: ID does not exist" containerID="b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"
Oct 30 12:03:34.505364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.505255    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038"} err="failed to get container status \"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038\": rpc error: code = NotFound desc = could not find container \"b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038\": container with ID starting with b6740e3084a6dc705169e83860a49f2e8a7108f636ec40342fd63297addfc038 not found: ID does not exist"
Oct 30 12:03:34.550721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.550695    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 12:03:34.560436 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.560410    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6tssr"]
Oct 30 12:03:34.911499 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:03:34.911468    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" path="/var/lib/kubelet/pods/27f99652-ea34-4cb3-ad33-55471d784e9b/volumes"
Oct 30 12:03:37.655761 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:03:37.655716    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:04:07.404826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.404777    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:04:07.405380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.405366    4453 cpu_manager.go:414] "containerMap: removing container" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy"
Oct 30 12:04:07.405380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.405380    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy"
Oct 30 12:04:07.405517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.405508    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f99652-ea34-4cb3-ad33-55471d784e9b" containerName="istio-proxy"
Oct 30 12:04:07.410216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.410194    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.433416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.433384    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:04:07.492594 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492562    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xd5\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492644    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492747    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492785    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492848    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492892    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.492956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492915    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.493058 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.492966    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.493118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.493079    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.571735 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:04:07.571698    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:04:07.594153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594117    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594213    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xd5\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594307    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594343    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594372    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594413    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594435    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594465    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594596    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594794    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.594838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.594808    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.595041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.595007    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.595185 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.595157    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.596714 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.596693    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.596840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.596823    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.615393 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.615369    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xd5\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.615551 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.615526    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\") pod \"e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") " pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:07.724227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:07.724137    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:08.041836 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.041802    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:04:08.043148 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:04:08.043114    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63607f10_f85d_449c_ba61_f3c20e520d52.slice/crio-ed77936b2eb5cb4ed4dea8dcf57470e8fa91641425a5974859d472b14dfa2649 WatchSource:0}: Error finding container ed77936b2eb5cb4ed4dea8dcf57470e8fa91641425a5974859d472b14dfa2649: Status 404 returned error can't find the container with id ed77936b2eb5cb4ed4dea8dcf57470e8fa91641425a5974859d472b14dfa2649
Oct 30 12:04:08.045611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.045572    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:04:08.045697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.045643    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:04:08.045697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.045672    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:04:08.878542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.878503    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" event={"ID":"63607f10-f85d-449c-ba61-f3c20e520d52","Type":"ContainerStarted","Data":"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"}
Oct 30 12:04:08.879006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.878561    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" event={"ID":"63607f10-f85d-449c-ba61-f3c20e520d52","Type":"ContainerStarted","Data":"ed77936b2eb5cb4ed4dea8dcf57470e8fa91641425a5974859d472b14dfa2649"}
Oct 30 12:04:08.962667 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:08.962608    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podStartSLOduration=1.9625940339999999 podStartE2EDuration="1.962594034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:04:08.950816664 +0000 UTC m=+2048.171101574" watchObservedRunningTime="2026-10-30 12:04:08.962594034 +0000 UTC m=+2048.182878943"
Oct 30 12:04:09.725104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:09.725069    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:09.726877 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:09.726856    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:09.726942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:09.726902    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:10.725501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:10.725463    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:10.725924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:10.725522    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:11.724742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:11.724707    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:11.724917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:11.724770    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:12.724996 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:12.724949    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:12.725489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:12.725002    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:13.724507 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:13.724472    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:13.724702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:13.724527    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:14.724887 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:14.724842    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 12:04:14.725473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:14.724893    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 12:04:15.775368 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:15.775305    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:15.775855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:15.775786    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:15.776720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:15.776701    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:04:35.980824 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.980790    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:04:35.981347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.981049    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy" containerID="cri-o://9a8e60bb2fdfcadc351d1dc064bc9b275dffca47955b7d89922c59e75266a173" gracePeriod=30
Oct 30 12:04:35.990560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.990534    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 12:04:35.990873 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.990780    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy" containerID="cri-o://9d253565f6b009232db40ae81f8b262f1bf31df3394eb88dc12b9eb4a79eaefc" gracePeriod=30
Oct 30 12:04:35.996495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.996470    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:04:35.996692 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:35.996671    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy" containerID="cri-o://3aff1f58abd22e4b8ba628c217e9359911b923dc6ed9cac2ce45f0d904edcde2" gracePeriod=30
Oct 30 12:04:37.570504 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:04:37.570458    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:04:41.252008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.251967    4453 generic.go:356] "Generic (PLEG): container finished" podID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerID="9d253565f6b009232db40ae81f8b262f1bf31df3394eb88dc12b9eb4a79eaefc" exitCode=0
Oct 30 12:04:41.252625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.252057    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" event={"ID":"17b18771-e8ec-4556-adf9-10d2ac283c48","Type":"ContainerDied","Data":"9d253565f6b009232db40ae81f8b262f1bf31df3394eb88dc12b9eb4a79eaefc"}
Oct 30 12:04:41.254808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.254778    4453 generic.go:356] "Generic (PLEG): container finished" podID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerID="9a8e60bb2fdfcadc351d1dc064bc9b275dffca47955b7d89922c59e75266a173" exitCode=0
Oct 30 12:04:41.254912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.254864    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" event={"ID":"6dbe00a0-6044-4618-9fe6-de23e959b0af","Type":"ContainerDied","Data":"9a8e60bb2fdfcadc351d1dc064bc9b275dffca47955b7d89922c59e75266a173"}
Oct 30 12:04:41.256942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.256913    4453 generic.go:356] "Generic (PLEG): container finished" podID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerID="3aff1f58abd22e4b8ba628c217e9359911b923dc6ed9cac2ce45f0d904edcde2" exitCode=0
Oct 30 12:04:41.257048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.256942    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" event={"ID":"1cb42af7-0b56-4a24-9ac6-a86c756b25cc","Type":"ContainerDied","Data":"3aff1f58abd22e4b8ba628c217e9359911b923dc6ed9cac2ce45f0d904edcde2"}
Oct 30 12:04:41.324093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.324063    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:04:41.349727 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.349707    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 12:04:41.360097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.360078    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:04:41.394447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394422    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394505    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394541    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394695 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394589    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394759 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394715    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394815 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394784    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394815 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394801    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.394924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394840    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.394924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394878    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.395041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394928    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.395041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.394962    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\") pod \"6dbe00a0-6044-4618-9fe6-de23e959b0af\" (UID: \"6dbe00a0-6044-4618-9fe6-de23e959b0af\") "
Oct 30 12:04:41.395191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395085    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.395286 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395259    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.395389 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395367    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.395641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395621    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dbe00a0-6044-4618-9fe6-de23e959b0af-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.395717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395646    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.395717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395661    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.395717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395677    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.395717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.395692    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.397431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.397401    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "kube-api-access-jjp2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.397635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.397616    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.398059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.398039    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.398386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.398366    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token" pod "6dbe00a0-6044-4618-9fe6-de23e959b0af" (UID: "6dbe00a0-6044-4618-9fe6-de23e959b0af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.496119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496081    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.496311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496161    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.496311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496194    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496243    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496270    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.496311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496301    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496366    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496408    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496471    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496507    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496508    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496543    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496575    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.496634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496603    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.497124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496637    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.497124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496710    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.497124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496802    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.497124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496835    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\") pod \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\" (UID: \"1cb42af7-0b56-4a24-9ac6-a86c756b25cc\") "
Oct 30 12:04:41.497124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.496902    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\") pod \"17b18771-e8ec-4556-adf9-10d2ac283c48\" (UID: \"17b18771-e8ec-4556-adf9-10d2ac283c48\") "
Oct 30 12:04:41.497573 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.497549    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.497662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.497579    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjp2t\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-kube-api-access-jjp2t\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.497662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.497595    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.497662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.497610    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dbe00a0-6044-4618-9fe6-de23e959b0af-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.497662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.497622    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.498629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.498599    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499110 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499081    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499115    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499158    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.499364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499324    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499384    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499434    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.499712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499572    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.500012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.499992    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.500528 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.500506    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:04:41.500902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.500882    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.501803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.501727    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.504807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.502216    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.504807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.502244    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "kube-api-access-k9xhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.504807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.502532    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:04:41.504807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.503581    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo" pod "1cb42af7-0b56-4a24-9ac6-a86c756b25cc" (UID: "1cb42af7-0b56-4a24-9ac6-a86c756b25cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:04:41.504807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.504186    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962" pod "17b18771-e8ec-4556-adf9-10d2ac283c48" (UID: "17b18771-e8ec-4556-adf9-10d2ac283c48"). InnerVolumeSpecName "kube-api-access-xz962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:04:41.599783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599747    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.599783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599780    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.599783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599789    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599798    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599806    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599814    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599822    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599834    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599848    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz962\" (UniqueName: \"kubernetes.io/projected/17b18771-e8ec-4556-adf9-10d2ac283c48-kube-api-access-xz962\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599857    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599866    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599874    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599882    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599890    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17b18771-e8ec-4556-adf9-10d2ac283c48-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599897    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xhf\" (UniqueName: \"kubernetes.io/projected/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-kube-api-access-k9xhf\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599908    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb42af7-0b56-4a24-9ac6-a86c756b25cc-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:41.600052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:41.599921    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17b18771-e8ec-4556-adf9-10d2ac283c48-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:04:42.264214 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.264183    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch" event={"ID":"17b18771-e8ec-4556-adf9-10d2ac283c48","Type":"ContainerDied","Data":"772cde8607b24422a18dd5b9a19c19ef6f166ef6f041c06d883e712047cef656"}
Oct 30 12:04:42.264659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.264195    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"
Oct 30 12:04:42.264659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.264240    4453 scope.go:122] "RemoveContainer" containerID="9d253565f6b009232db40ae81f8b262f1bf31df3394eb88dc12b9eb4a79eaefc"
Oct 30 12:04:42.265710 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.265691    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj" event={"ID":"6dbe00a0-6044-4618-9fe6-de23e959b0af","Type":"ContainerDied","Data":"b2ab7cf108e29d9c52851577e6809d0fa69896428d7460c74875c51673f1ec21"}
Oct 30 12:04:42.265780 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.265772    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"
Oct 30 12:04:42.270124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.270103    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"
Oct 30 12:04:42.270213 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.270107    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm" event={"ID":"1cb42af7-0b56-4a24-9ac6-a86c756b25cc","Type":"ContainerDied","Data":"88651f49e577907cb47a3d961d38bad45fa4b538315f24ce57458453df28ab07"}
Oct 30 12:04:42.276079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.276059    4453 scope.go:122] "RemoveContainer" containerID="9a8e60bb2fdfcadc351d1dc064bc9b275dffca47955b7d89922c59e75266a173"
Oct 30 12:04:42.286132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.286113    4453 scope.go:122] "RemoveContainer" containerID="3aff1f58abd22e4b8ba628c217e9359911b923dc6ed9cac2ce45f0d904edcde2"
Oct 30 12:04:42.336033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.336007    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:04:42.345300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.345276    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-fqbhj"]
Oct 30 12:04:42.372678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.372651    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 12:04:42.379085 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.379059    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-8n9ch"]
Oct 30 12:04:42.408829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.408804    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:04:42.416697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.416672    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e25f4a-openshift-default-d4446c48f-x5qkm"]
Oct 30 12:04:42.912256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.912224    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" path="/var/lib/kubelet/pods/17b18771-e8ec-4556-adf9-10d2ac283c48/volumes"
Oct 30 12:04:42.912731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.912716    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" path="/var/lib/kubelet/pods/1cb42af7-0b56-4a24-9ac6-a86c756b25cc/volumes"
Oct 30 12:04:42.913145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:04:42.913133    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" path="/var/lib/kubelet/pods/6dbe00a0-6044-4618-9fe6-de23e959b0af/volumes"
Oct 30 12:05:07.570672 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:05:07.570634    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:05:26.380285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380202    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:05:26.380832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380815    4453 cpu_manager.go:414] "containerMap: removing container" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy"
Oct 30 12:05:26.380879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380833    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy"
Oct 30 12:05:26.380879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380856    4453 cpu_manager.go:414] "containerMap: removing container" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy"
Oct 30 12:05:26.380879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380862    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy"
Oct 30 12:05:26.380879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380870    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy"
Oct 30 12:05:26.380879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380875    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy"
Oct 30 12:05:26.381056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380967    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b18771-e8ec-4556-adf9-10d2ac283c48" containerName="istio-proxy"
Oct 30 12:05:26.381056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380981    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbe00a0-6044-4618-9fe6-de23e959b0af" containerName="istio-proxy"
Oct 30 12:05:26.381056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.380990    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb42af7-0b56-4a24-9ac6-a86c756b25cc" containerName="istio-proxy"
Oct 30 12:05:26.384617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.384600    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.399115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.399077    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4fff4f-dockercfg-bjmbv\""
Oct 30 12:05:26.419993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.419956    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:05:26.495698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.495662    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4fff4f\""
Oct 30 12:05:26.524565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.524520    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.524920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.524897    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqvv\" (UniqueName: \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.626025 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.625989    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqvv\" (UniqueName: \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.626238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.626195    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.627188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.627155    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.647927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.647854    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqvv\" (UniqueName: \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\") pod \"payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:26.700037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:26.700007    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:27.004953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:27.004917    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:05:27.005801 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:05:27.005763    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e04b890_36bb_4f10_ac03_00b75d288c08.slice/crio-68776148589e8644a2b8ff510e5af4941a59a8090d44fb32bfc20f41444c8460 WatchSource:0}: Error finding container 68776148589e8644a2b8ff510e5af4941a59a8090d44fb32bfc20f41444c8460: Status 404 returned error can't find the container with id 68776148589e8644a2b8ff510e5af4941a59a8090d44fb32bfc20f41444c8460
Oct 30 12:05:27.614260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:27.614226    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" event={"ID":"0e04b890-36bb-4f10-ac03-00b75d288c08","Type":"ContainerStarted","Data":"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"}
Oct 30 12:05:27.614686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:27.614278    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" event={"ID":"0e04b890-36bb-4f10-ac03-00b75d288c08","Type":"ContainerStarted","Data":"68776148589e8644a2b8ff510e5af4941a59a8090d44fb32bfc20f41444c8460"}
Oct 30 12:05:27.614686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:27.614382    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:27.659944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:27.659887    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" podStartSLOduration=1.6598743219999998 podStartE2EDuration="1.659874322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:27.658221071 +0000 UTC m=+2126.878505981" watchObservedRunningTime="2026-10-30 12:05:27.659874322 +0000 UTC m=+2126.880159267"
Oct 30 12:05:35.478512 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.478437    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:05:35.482807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.482785    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.535288 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.535252    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:05:35.566406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566360    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566581 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566520    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7dvz\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566639    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566672    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566818 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566782    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566818    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566912    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.566991 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.566945    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.567106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.567080    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.588846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.588826    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-28ca05-openshift-default-dockercfg-9kwzs\""
Oct 30 12:05:35.667775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.667741    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7dvz\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.667938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.667842    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.667938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.667875    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.667943    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.667978    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668040    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668100    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668187    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668407    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668422    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668582    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.668814 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.668703    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.669031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.669010    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.669123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.669062    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.670572 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.670550    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.670967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.670940    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.697862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.697835    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.700263 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.700227    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7dvz\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:35.799043 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:35.798944    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:36.152628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.152600    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:05:36.153662 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:05:36.153638    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9da1c2_b651_4fec_bf7b_a11e6a407199.slice/crio-a20c39475abdd909a1545f2cc34d447f2fb2a1cd5a9e199a9f13b68755e4ee34 WatchSource:0}: Error finding container a20c39475abdd909a1545f2cc34d447f2fb2a1cd5a9e199a9f13b68755e4ee34: Status 404 returned error can't find the container with id a20c39475abdd909a1545f2cc34d447f2fb2a1cd5a9e199a9f13b68755e4ee34
Oct 30 12:05:36.156253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.156214    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:05:36.156385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.156298    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:05:36.156385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.156358    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:05:36.669181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.669140    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" event={"ID":"dc9da1c2-b651-4fec-bf7b-a11e6a407199","Type":"ContainerStarted","Data":"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"}
Oct 30 12:05:36.669609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.669207    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" event={"ID":"dc9da1c2-b651-4fec-bf7b-a11e6a407199","Type":"ContainerStarted","Data":"a20c39475abdd909a1545f2cc34d447f2fb2a1cd5a9e199a9f13b68755e4ee34"}
Oct 30 12:05:36.750976 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.750923    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podStartSLOduration=1.750908426 podStartE2EDuration="1.750908426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:05:36.740959115 +0000 UTC m=+2135.961244025" watchObservedRunningTime="2026-10-30 12:05:36.750908426 +0000 UTC m=+2135.971193335"
Oct 30 12:05:36.808469 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:36.808423    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:37.571103 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:05:37.571063    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:05:37.814496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:37.814462    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 12:05:37.815086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:37.815039    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 12:05:38.623259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:38.623222    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:05:38.813842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:38.813790    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 12:05:38.814049 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:38.813871    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 12:05:39.813807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:39.813769    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 12:05:39.814420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:39.813831    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 12:05:40.810817 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:40.810775    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 12:05:40.811231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:40.811198    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 12:05:41.813808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:41.813766    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 12:05:41.814221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:41.813869    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 12:05:42.811710 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:42.811672    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:42.812177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:42.812151    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:05:42.812967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:05:42.812947    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:06:07.608382 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:06:07.600090    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:06:20.602077 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.602039    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:06:20.608705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.608680    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.639196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.639160    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:06:20.722813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.722777    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.722813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.722816    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.722874    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.722898    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.723130    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.723191    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.723301    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.723414    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.723463 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.723444    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pjk\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824074    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824149    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824169    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pjk\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824209    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824231    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824299    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824381    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824440    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824577    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824835 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824741    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824835 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824768    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.824916 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.824901    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.825180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.825157    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.826720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.826683    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.826982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.826964    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.844781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.844759    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.845207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.845187    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pjk\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\") pod \"e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") " pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:20.927537 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:20.927510    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:21.248500 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:06:21.248463    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79716152_156c_4d1e_9fc2_465dc9e6995c.slice/crio-ab7bbb8f19e58a523a621c0cab489f73c05a106b9ce49ff4ae6f8ce22d49d3ce WatchSource:0}: Error finding container ab7bbb8f19e58a523a621c0cab489f73c05a106b9ce49ff4ae6f8ce22d49d3ce: Status 404 returned error can't find the container with id ab7bbb8f19e58a523a621c0cab489f73c05a106b9ce49ff4ae6f8ce22d49d3ce
Oct 30 12:06:21.248647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:21.248529    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:06:21.251143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:21.251114    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:06:21.251225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:21.251177    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:06:21.251225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:21.251206    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:06:22.028221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.028185    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" event={"ID":"79716152-156c-4d1e-9fc2-465dc9e6995c","Type":"ContainerStarted","Data":"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"}
Oct 30 12:06:22.028689 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.028242    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" event={"ID":"79716152-156c-4d1e-9fc2-465dc9e6995c","Type":"ContainerStarted","Data":"ab7bbb8f19e58a523a621c0cab489f73c05a106b9ce49ff4ae6f8ce22d49d3ce"}
Oct 30 12:06:22.082658 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.082598    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podStartSLOduration=2.082582333 podStartE2EDuration="2.082582333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:22.080390807 +0000 UTC m=+2181.300675719" watchObservedRunningTime="2026-10-30 12:06:22.082582333 +0000 UTC m=+2181.302867247"
Oct 30 12:06:22.927965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.927932    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:22.929667 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.929641    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:06:22.929824 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:22.929687    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:06:23.858354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.857985    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:06:23.917354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.913549    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:06:23.917354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.914651    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:23.933354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.933009    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-4a92de\""
Oct 30 12:06:23.943998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.933583    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:06:23.943998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.933639    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:06:23.956470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.956438    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-4a92de-dockercfg-4c8bj\""
Oct 30 12:06:23.983957 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.983917    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:23.984562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:23.984522    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jh8\" (UniqueName: \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.085795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.085759    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jh8\" (UniqueName: \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.085988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.085925    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.086709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.086678    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.120218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.120141    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jh8\" (UniqueName: \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\") pod \"payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.269762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.269730    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:24.622531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.622504    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:06:24.625013 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:06:24.624980    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec90dfa7_8dc3_4928_a13b_26571018407f.slice/crio-fe6502aaa17e544ae94d1acf1d1541a890aa21feccb9cce602781dfd76795b97 WatchSource:0}: Error finding container fe6502aaa17e544ae94d1acf1d1541a890aa21feccb9cce602781dfd76795b97: Status 404 returned error can't find the container with id fe6502aaa17e544ae94d1acf1d1541a890aa21feccb9cce602781dfd76795b97
Oct 30 12:06:24.928471 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.928379    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:06:24.928471 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:24.928434    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:06:25.050728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.050691    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" event={"ID":"ec90dfa7-8dc3-4928-a13b-26571018407f","Type":"ContainerStarted","Data":"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"}
Oct 30 12:06:25.050911 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.050741    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" event={"ID":"ec90dfa7-8dc3-4928-a13b-26571018407f","Type":"ContainerStarted","Data":"fe6502aaa17e544ae94d1acf1d1541a890aa21feccb9cce602781dfd76795b97"}
Oct 30 12:06:25.050911 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.050784    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:25.112546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.107385    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" podStartSLOduration=2.107361652 podStartE2EDuration="2.107361652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:25.099694538 +0000 UTC m=+2184.319979448" watchObservedRunningTime="2026-10-30 12:06:25.107361652 +0000 UTC m=+2184.327646564"
Oct 30 12:06:25.928432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.928395    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:06:25.928609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:25.928454    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:06:26.943483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:26.942756    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 12:06:26.943483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:26.942833    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 12:06:27.933005 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:27.932976    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:27.933464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:27.933443    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:27.934209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:27.934192    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:06:36.060734 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:36.060704    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:06:37.570508 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:06:37.570472    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:06:57.920709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:57.920625    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:06:57.924564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:57.924547    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:57.937837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:57.937807    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a51616-dockercfg-hrn8t\""
Oct 30 12:06:57.947548 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:57.947522    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a51616\""
Oct 30 12:06:57.951148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:57.951125    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:06:58.028381 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.024396    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.028381 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.024666    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fphb\" (UniqueName: \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.134404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.130612    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fphb\" (UniqueName: \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.134404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.130782    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.143953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.143914    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.148781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.148246    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:06:58.164740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.164704    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:06:58.164979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.164799    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.181157 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.181093    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fphb\" (UniqueName: \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\") pod \"payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.232392 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.232364    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.232571 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.232547    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcnr\" (UniqueName: \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.237966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.237944    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:58.333934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.333882    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcnr\" (UniqueName: \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.334314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.334293    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.335166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.335143    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.355756 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.355725    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcnr\" (UniqueName: \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\") pod \"payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") " pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.496514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.496440    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:58.541725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.541692    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:06:58.542276 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:06:58.542252    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430f3db0_677b_435c_a5ce_96abeede3878.slice/crio-97d062ba07d28e45d0c457dfdf1ab33b4c2eec7b4bc9f2ac0ad3f16baa1e824b WatchSource:0}: Error finding container 97d062ba07d28e45d0c457dfdf1ab33b4c2eec7b4bc9f2ac0ad3f16baa1e824b: Status 404 returned error can't find the container with id 97d062ba07d28e45d0c457dfdf1ab33b4c2eec7b4bc9f2ac0ad3f16baa1e824b
Oct 30 12:06:58.813912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:58.813880    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:06:58.815185 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:06:58.815156    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d2bfe1_1dd4_4827_9249_7ea93532b888.slice/crio-64cf5a1c8a23c970f1c8ce8a8038e19cd5f4303a74d610b57138e1956e8f6512 WatchSource:0}: Error finding container 64cf5a1c8a23c970f1c8ce8a8038e19cd5f4303a74d610b57138e1956e8f6512: Status 404 returned error can't find the container with id 64cf5a1c8a23c970f1c8ce8a8038e19cd5f4303a74d610b57138e1956e8f6512
Oct 30 12:06:59.348525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.348408    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" event={"ID":"42d2bfe1-1dd4-4827-9249-7ea93532b888","Type":"ContainerStarted","Data":"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"}
Oct 30 12:06:59.348525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.348513    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" event={"ID":"42d2bfe1-1dd4-4827-9249-7ea93532b888","Type":"ContainerStarted","Data":"64cf5a1c8a23c970f1c8ce8a8038e19cd5f4303a74d610b57138e1956e8f6512"}
Oct 30 12:06:59.350436 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.350412    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:06:59.351397 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.351363    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" event={"ID":"430f3db0-677b-435c-a5ce-96abeede3878","Type":"ContainerStarted","Data":"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"}
Oct 30 12:06:59.351518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.351425    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" event={"ID":"430f3db0-677b-435c-a5ce-96abeede3878","Type":"ContainerStarted","Data":"97d062ba07d28e45d0c457dfdf1ab33b4c2eec7b4bc9f2ac0ad3f16baa1e824b"}
Oct 30 12:06:59.351518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.351466    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:06:59.388549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.388501    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" podStartSLOduration=2.388487152 podStartE2EDuration="2.388487152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:59.385169622 +0000 UTC m=+2218.605454532" watchObservedRunningTime="2026-10-30 12:06:59.388487152 +0000 UTC m=+2218.608772062"
Oct 30 12:06:59.421469 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:06:59.421426    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" podStartSLOduration=2.421415365 podStartE2EDuration="2.421415365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:06:59.420396627 +0000 UTC m=+2218.640681539" watchObservedRunningTime="2026-10-30 12:06:59.421415365 +0000 UTC m=+2218.641700275"
Oct 30 12:07:07.572034 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:07:07.571993    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:07:10.361079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:10.361048    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:07:11.366552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:11.366524    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:07:22.275819 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.275780    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:07:22.281359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.281314    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.294056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.294032    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-a51616-serving-cert\""
Oct 30 12:07:22.312229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.312194    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:07:22.378470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.378431    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.378658 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.378569    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshhn\" (UniqueName: \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.378725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.378653    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.378787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.378736    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.484244    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.484398    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshhn\" (UniqueName: \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.484479    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.484603    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.485586    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:07:22.486829    4453 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-serving-cert: secret "maas-api-e2e-explicit-a51616-serving-cert" not found
Oct 30 12:07:22.493353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:07:22.486895    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls podName:fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e nodeName:}" failed. No retries permitted until 2026-10-30 12:07:22.986875371 +0000 UTC m=+2242.207160259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls") pod "maas-api-e2e-explicit-a51616-75756f978d-n92jd" (UID: "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e") : secret "maas-api-e2e-explicit-a51616-serving-cert" not found
Oct 30 12:07:22.502866 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.502780    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.532671 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.532591    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshhn\" (UniqueName: \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.991534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.991501    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:22.994125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:22.994093    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") pod \"maas-api-e2e-explicit-a51616-75756f978d-n92jd\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:23.195216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:23.195179    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:23.497717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:23.497691    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:07:23.500549 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:07:23.500515    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa81c26f_8317_4c5f_8dfb_b9785ba9ad3e.slice/crio-7310dd86c4e2a4b9c106304f014bd817b83e7057890d1368ff861ced500f394d WatchSource:0}: Error finding container 7310dd86c4e2a4b9c106304f014bd817b83e7057890d1368ff861ced500f394d: Status 404 returned error can't find the container with id 7310dd86c4e2a4b9c106304f014bd817b83e7057890d1368ff861ced500f394d
Oct 30 12:07:23.502281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:23.502260    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:07:23.542090 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:23.542060    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" event={"ID":"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e","Type":"ContainerStarted","Data":"7310dd86c4e2a4b9c106304f014bd817b83e7057890d1368ff861ced500f394d"}
Oct 30 12:07:24.549997 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:24.549962    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" event={"ID":"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e","Type":"ContainerStarted","Data":"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"}
Oct 30 12:07:24.550449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:24.550149    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:24.588728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:24.588674    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" podStartSLOduration=2.224837754 podStartE2EDuration="2.588657861s" totalImagesPullingTime="363.820107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:24.586483129 +0000 UTC m=+2243.806768041" watchObservedRunningTime="2026-10-30 12:07:24.588657861 +0000 UTC m=+2243.808942770"
Oct 30 12:07:28.885631 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.885596    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:07:28.889684 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.889667    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.901781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.901756    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"kube-root-ca.crt\""
Oct 30 12:07:28.913044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.913016    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"auth-test-model-87f2e6-kserve-self-signed-certs\""
Oct 30 12:07:28.913361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.913311    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"default-dockercfg-7wlq2\""
Oct 30 12:07:28.913489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.913419    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-87f2e6\"/\"openshift-service-ca.crt\""
Oct 30 12:07:28.916045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.916013    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:07:28.970290 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970263    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.970473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970305    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.970516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970480    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.970603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970583    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.970639 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970625    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ltd\" (UniqueName: \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:28.970726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:28.970715    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.071601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071567    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.071806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071640    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.071806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071715    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.071931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071813    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.071931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071883    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ltd\" (UniqueName: \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.072042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071951    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.072042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.071968    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.072464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.072439    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.072625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.072600    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.074232 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.074208    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.074673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.074653    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.092649 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.092628    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ltd\" (UniqueName: \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\") pod \"auth-test-model-87f2e6-kserve-76bb657c89-dltdt\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") " pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.206736 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.206641    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:29.541170 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.541140    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:07:29.542773 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:07:29.542743    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac10d7e4_474c_4c44_92e0_55d04eeeb31e.slice/crio-ebd3f5d367549e650ecd8f0ba5d49d164a4971d4fa591082f9ffea889bed5cbf WatchSource:0}: Error finding container ebd3f5d367549e650ecd8f0ba5d49d164a4971d4fa591082f9ffea889bed5cbf: Status 404 returned error can't find the container with id ebd3f5d367549e650ecd8f0ba5d49d164a4971d4fa591082f9ffea889bed5cbf
Oct 30 12:07:29.583141 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:29.583113    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" event={"ID":"ac10d7e4-474c-4c44-92e0-55d04eeeb31e","Type":"ContainerStarted","Data":"ebd3f5d367549e650ecd8f0ba5d49d164a4971d4fa591082f9ffea889bed5cbf"}
Oct 30 12:07:30.561006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:30.560972    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:07:30.591264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:30.591223    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" event={"ID":"ac10d7e4-474c-4c44-92e0-55d04eeeb31e","Type":"ContainerStarted","Data":"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"}
Oct 30 12:07:30.634929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:30.634877    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" podStartSLOduration=2.634862241 podStartE2EDuration="2.634862241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:07:30.633079045 +0000 UTC m=+2249.853363977" watchObservedRunningTime="2026-10-30 12:07:30.634862241 +0000 UTC m=+2249.855147150"
Oct 30 12:07:31.207646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:31.207616    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:37.570461 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:07:37.570421    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:07:39.207896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:39.207865    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:39.220770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:39.220740    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:07:39.680061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:07:39.680031    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:08:07.599308 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:08:07.599265    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:08:20.176104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.176026    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:08:20.181009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.180984    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.205933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.205903    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"default-dockercfg-pkfsc\""
Oct 30 12:08:20.206262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.206236    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"auth-test-model-4a92de-kserve-self-signed-certs\""
Oct 30 12:08:20.210503 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.210480    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:08:20.215621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.215595    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"kube-root-ca.crt\""
Oct 30 12:08:20.216109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.216089    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-4a92de\"/\"openshift-service-ca.crt\""
Oct 30 12:08:20.290774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.290742    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.290954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.290897    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.291022 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.290992    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.291207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.291184    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.291285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.291237    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.291438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.291416    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28w6\" (UniqueName: \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393013 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.392973    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393080    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393152    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393235    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393290    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393430    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28w6\" (UniqueName: \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393598    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393796 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393703    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.393913 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.393880    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.395563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.395538    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.395830 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.395813    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.414934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.414911    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28w6\" (UniqueName: \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\") pod \"auth-test-model-4a92de-kserve-77bf54676f-qd6fv\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") " pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.495916 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.495823    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:20.809367 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:08:20.809296    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb91fd83_4956_458b_8805_9153ef072e25.slice/crio-aba04dedce90c516af475a9af7b2fa8c9d7000f14a6931d564d8cba103197563 WatchSource:0}: Error finding container aba04dedce90c516af475a9af7b2fa8c9d7000f14a6931d564d8cba103197563: Status 404 returned error can't find the container with id aba04dedce90c516af475a9af7b2fa8c9d7000f14a6931d564d8cba103197563
Oct 30 12:08:20.809367 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.809319    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:08:20.982982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.982943    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" event={"ID":"eb91fd83-4956-458b-8805-9153ef072e25","Type":"ContainerStarted","Data":"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"}
Oct 30 12:08:20.983151 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:20.982990    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" event={"ID":"eb91fd83-4956-458b-8805-9153ef072e25","Type":"ContainerStarted","Data":"aba04dedce90c516af475a9af7b2fa8c9d7000f14a6931d564d8cba103197563"}
Oct 30 12:08:21.029249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:21.029201    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" podStartSLOduration=1.029185779 podStartE2EDuration="1.029185779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:08:21.018081724 +0000 UTC m=+2300.238366654" watchObservedRunningTime="2026-10-30 12:08:21.029185779 +0000 UTC m=+2300.249470688"
Oct 30 12:08:21.496324 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:21.496289    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:30.496309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:30.496271    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:30.509059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:30.509032    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:31.055165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:08:31.055135    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:08:37.570706 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:08:37.570671    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:09:02.379760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:02.379728    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:09:02.413454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:02.413423    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:09:07.380045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.379989    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" podUID="42d2bfe1-1dd4-4827-9249-7ea93532b888" containerName="payload-processing" containerID="cri-o://042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086" gracePeriod=25
Oct 30 12:09:07.414265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.414230    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" podUID="430f3db0-677b-435c-a5ce-96abeede3878" containerName="payload-pre-processing" containerID="cri-o://74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9" gracePeriod=25
Oct 30 12:09:07.574278 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:07.574214    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:09:07.695293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.695266    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:09:07.728404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.728383    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:09:07.795974 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.795940    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\") pod \"42d2bfe1-1dd4-4827-9249-7ea93532b888\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") "
Oct 30 12:09:07.796190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.796166    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\" (UniqueName: \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\") pod \"42d2bfe1-1dd4-4827-9249-7ea93532b888\" (UID: \"42d2bfe1-1dd4-4827-9249-7ea93532b888\") "
Oct 30 12:09:07.796387 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.796361    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume" pod "42d2bfe1-1dd4-4827-9249-7ea93532b888" (UID: "42d2bfe1-1dd4-4827-9249-7ea93532b888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:07.796815 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.796793    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d2bfe1-1dd4-4827-9249-7ea93532b888-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:07.798292 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.798269    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr" pod "42d2bfe1-1dd4-4827-9249-7ea93532b888" (UID: "42d2bfe1-1dd4-4827-9249-7ea93532b888"). InnerVolumeSpecName "kube-api-access-8jcnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:07.899895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.898640    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\" (UniqueName: \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\") pod \"430f3db0-677b-435c-a5ce-96abeede3878\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") "
Oct 30 12:09:07.899895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.898853    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\") pod \"430f3db0-677b-435c-a5ce-96abeede3878\" (UID: \"430f3db0-677b-435c-a5ce-96abeede3878\") "
Oct 30 12:09:07.899895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.899598    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jcnr\" (UniqueName: \"kubernetes.io/projected/42d2bfe1-1dd4-4827-9249-7ea93532b888-kube-api-access-8jcnr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:07.900324 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.900299    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume" pod "430f3db0-677b-435c-a5ce-96abeede3878" (UID: "430f3db0-677b-435c-a5ce-96abeede3878"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:07.905520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:07.905486    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb" pod "430f3db0-677b-435c-a5ce-96abeede3878" (UID: "430f3db0-677b-435c-a5ce-96abeede3878"). InnerVolumeSpecName "kube-api-access-7fphb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:08.000711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.000674    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fphb\" (UniqueName: \"kubernetes.io/projected/430f3db0-677b-435c-a5ce-96abeede3878-kube-api-access-7fphb\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:08.000711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.000705    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430f3db0-677b-435c-a5ce-96abeede3878-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:08.344168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.344085    4453 generic.go:356] "Generic (PLEG): container finished" podID="430f3db0-677b-435c-a5ce-96abeede3878" containerID="74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9" exitCode=0
Oct 30 12:09:08.344168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.344131    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"
Oct 30 12:09:08.344445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.344173    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" event={"ID":"430f3db0-677b-435c-a5ce-96abeede3878","Type":"ContainerDied","Data":"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"}
Oct 30 12:09:08.344445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.344224    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x" event={"ID":"430f3db0-677b-435c-a5ce-96abeede3878","Type":"ContainerDied","Data":"97d062ba07d28e45d0c457dfdf1ab33b4c2eec7b4bc9f2ac0ad3f16baa1e824b"}
Oct 30 12:09:08.344445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.344246    4453 scope.go:122] "RemoveContainer" containerID="74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"
Oct 30 12:09:08.345735 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.345714    4453 generic.go:356] "Generic (PLEG): container finished" podID="42d2bfe1-1dd4-4827-9249-7ea93532b888" containerID="042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086" exitCode=0
Oct 30 12:09:08.345851 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.345745    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" event={"ID":"42d2bfe1-1dd4-4827-9249-7ea93532b888","Type":"ContainerDied","Data":"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"}
Oct 30 12:09:08.345851 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.345765    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"
Oct 30 12:09:08.345851 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.345791    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs" event={"ID":"42d2bfe1-1dd4-4827-9249-7ea93532b888","Type":"ContainerDied","Data":"64cf5a1c8a23c970f1c8ce8a8038e19cd5f4303a74d610b57138e1956e8f6512"}
Oct 30 12:09:08.358016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.357998    4453 scope.go:122] "RemoveContainer" containerID="74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"
Oct 30 12:09:08.358354 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:08.358300    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9\": container with ID starting with 74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9 not found: ID does not exist" containerID="74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"
Oct 30 12:09:08.358448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.358371    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9"} err="failed to get container status \"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9\": rpc error: code = NotFound desc = could not find container \"74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9\": container with ID starting with 74e277698dfd767430d081b660d431aeea6c667266b29e720c2a8b2e22851eb9 not found: ID does not exist"
Oct 30 12:09:08.358448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.358390    4453 scope.go:122] "RemoveContainer" containerID="042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"
Oct 30 12:09:08.367146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.367129    4453 scope.go:122] "RemoveContainer" containerID="042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"
Oct 30 12:09:08.367400 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:08.367382    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086\": container with ID starting with 042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086 not found: ID does not exist" containerID="042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"
Oct 30 12:09:08.367453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.367406    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086"} err="failed to get container status \"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086\": rpc error: code = NotFound desc = could not find container \"042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086\": container with ID starting with 042f9734442c8e97471e0bfb64359217f99363d5f43b37cb76601c860c420086 not found: ID does not exist"
Oct 30 12:09:08.400978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.400958    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:09:08.410216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.410197    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a51616-cb69f88f4-mwffs"]
Oct 30 12:09:08.432779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.432759    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:09:08.441981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.441959    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a51616-8bb96fd47-rz67x"]
Oct 30 12:09:08.913089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.913050    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d2bfe1-1dd4-4827-9249-7ea93532b888" path="/var/lib/kubelet/pods/42d2bfe1-1dd4-4827-9249-7ea93532b888/volumes"
Oct 30 12:09:08.913560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:08.913537    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430f3db0-677b-435c-a5ce-96abeede3878" path="/var/lib/kubelet/pods/430f3db0-677b-435c-a5ce-96abeede3878/volumes"
Oct 30 12:09:09.239794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.239197    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:09:09.239794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.239508    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" podUID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" containerName="maas-api" containerID="cri-o://20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0" gracePeriod=30
Oct 30 12:09:09.594673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.594641    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:09:09.624255 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624227    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\") pod \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") "
Oct 30 12:09:09.624501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624390    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") pod \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") "
Oct 30 12:09:09.624501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624426    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\") pod \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") "
Oct 30 12:09:09.624501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624458    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\" (UniqueName: \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\") pod \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\" (UID: \"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e\") "
Oct 30 12:09:09.624702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624658    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca" pod "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" (UID: "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:09.625013 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.624992    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:09.626686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.626659    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs" pod "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" (UID: "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:09:09.626777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.626749    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls" pod "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" (UID: "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:09:09.626854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.626830    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn" pod "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" (UID: "fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e"). InnerVolumeSpecName "kube-api-access-cshhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:09.737351 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.734390    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:09.737351 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.734427    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:09.737351 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:09.734442    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshhn\" (UniqueName: \"kubernetes.io/projected/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e-kube-api-access-cshhn\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:10.361232 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.361195    4453 generic.go:356] "Generic (PLEG): container finished" podID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" containerID="20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0" exitCode=0
Oct 30 12:09:10.361432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.361250    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"
Oct 30 12:09:10.361432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.361280    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" event={"ID":"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e","Type":"ContainerDied","Data":"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"}
Oct 30 12:09:10.361432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.361346    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd" event={"ID":"fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e","Type":"ContainerDied","Data":"7310dd86c4e2a4b9c106304f014bd817b83e7057890d1368ff861ced500f394d"}
Oct 30 12:09:10.361432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.361370    4453 scope.go:122] "RemoveContainer" containerID="20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"
Oct 30 12:09:10.371553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.371536    4453 scope.go:122] "RemoveContainer" containerID="20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"
Oct 30 12:09:10.371823 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:10.371803    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0\": container with ID starting with 20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0 not found: ID does not exist" containerID="20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"
Oct 30 12:09:10.371875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.371831    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0"} err="failed to get container status \"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0\": rpc error: code = NotFound desc = could not find container \"20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0\": container with ID starting with 20e1727f53e615c7558bb0dca14e514675fc29f75fb044ed2eebae3a85a728d0 not found: ID does not exist"
Oct 30 12:09:10.406254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.406224    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:09:10.415462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.415440    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a51616-75756f978d-n92jd"]
Oct 30 12:09:10.912742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:10.912709    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" path="/var/lib/kubelet/pods/fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e/volumes"
Oct 30 12:09:18.892164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.892129    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:18.893087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893059    4453 cpu_manager.go:414] "containerMap: removing container" podUID="42d2bfe1-1dd4-4827-9249-7ea93532b888" containerName="payload-processing"
Oct 30 12:09:18.893087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893086    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d2bfe1-1dd4-4827-9249-7ea93532b888" containerName="payload-processing"
Oct 30 12:09:18.893293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893101    4453 cpu_manager.go:414] "containerMap: removing container" podUID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" containerName="maas-api"
Oct 30 12:09:18.893293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893111    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" containerName="maas-api"
Oct 30 12:09:18.893293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893149    4453 cpu_manager.go:414] "containerMap: removing container" podUID="430f3db0-677b-435c-a5ce-96abeede3878" containerName="payload-pre-processing"
Oct 30 12:09:18.893293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893157    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430f3db0-677b-435c-a5ce-96abeede3878" containerName="payload-pre-processing"
Oct 30 12:09:18.893293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893279    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="430f3db0-677b-435c-a5ce-96abeede3878" containerName="payload-pre-processing"
Oct 30 12:09:18.893561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893299    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d2bfe1-1dd4-4827-9249-7ea93532b888" containerName="payload-processing"
Oct 30 12:09:18.893561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.893308    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa81c26f-8317-4c5f-8dfb-b9785ba9ad3e" containerName="maas-api"
Oct 30 12:09:18.898059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.898039    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.914863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.913178    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-afbe5f-openshift-default-dockercfg-tt4qv\""
Oct 30 12:09:18.930619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.930574    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:18.961175 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961133    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961175 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961173    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961198    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961389    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961440    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961581 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961564    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961632    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbt8\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961668    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:18.961766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:18.961742    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063216    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063278    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbt8\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063315    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063366    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063601    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063631    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063657    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063709    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063741    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063778    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.063987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.063929    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.064112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.064091    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.064238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.064146    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.064238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.064195    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.065682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.065656    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.066199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.066171    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.084237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.084209    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.084884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.084860    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbt8\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\") pod \"e2e-noten-afbe5f-openshift-default-666c6686c5-82b59\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") " pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.239415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.238753    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:19.607595 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.607175    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:19.609712 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:09:19.609680    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e138d4_0462_461d_b3e7_5f30668330b3.slice/crio-465811bfec723709e86baea5caaff3c732510b2ac5985b547790290fa760e898 WatchSource:0}: Error finding container 465811bfec723709e86baea5caaff3c732510b2ac5985b547790290fa760e898: Status 404 returned error can't find the container with id 465811bfec723709e86baea5caaff3c732510b2ac5985b547790290fa760e898
Oct 30 12:09:19.612501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.612466    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:09:19.612578 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.612539    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:09:19.612578 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:19.612568    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:09:20.427688 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:20.427654    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" event={"ID":"78e138d4-0462-461d-b3e7-5f30668330b3","Type":"ContainerStarted","Data":"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"}
Oct 30 12:09:20.428186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:20.427708    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" event={"ID":"78e138d4-0462-461d-b3e7-5f30668330b3","Type":"ContainerStarted","Data":"465811bfec723709e86baea5caaff3c732510b2ac5985b547790290fa760e898"}
Oct 30 12:09:20.487895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:20.487838    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" podStartSLOduration=2.487821448 podStartE2EDuration="2.487821448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:09:20.477106811 +0000 UTC m=+2359.697391721" watchObservedRunningTime="2026-10-30 12:09:20.487821448 +0000 UTC m=+2359.708106408"
Oct 30 12:09:21.239820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:21.239784    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:21.246678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:21.246650    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:21.435231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:21.435187    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:21.436502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:21.436476    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:36.996946 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:36.996912    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:09:36.997481 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:36.997126    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy" containerID="cri-o://cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387" gracePeriod=30
Oct 30 12:09:37.570398 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:37.570364    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:09:42.272748 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.272686    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:09:42.307095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307060    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307163    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307206    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307309    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.307411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307364    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307409    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307440    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307486    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307554    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.307681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307596    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:42.307681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307632    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.307681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307637    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.307889 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.307788    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\") pod \"43388545-8b63-4978-a970-19a398a270b6\" (UID: \"43388545-8b63-4978-a970-19a398a270b6\") "
Oct 30 12:09:42.308107 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308081    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.308461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308440    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.308461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308459    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.308626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308468    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43388545-8b63-4978-a970-19a398a270b6-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.308626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308477    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.308626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.308487    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.309826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.309801    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:42.309933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.309865    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:09:42.310103 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.310075    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "kube-api-access-5ggp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:42.310168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.310104    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy" pod "43388545-8b63-4978-a970-19a398a270b6" (UID: "43388545-8b63-4978-a970-19a398a270b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:42.410167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.410135    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggp9\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-kube-api-access-5ggp9\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.410167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.410164    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43388545-8b63-4978-a970-19a398a270b6-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.410167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.410173    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43388545-8b63-4978-a970-19a398a270b6-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.410419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.410182    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43388545-8b63-4978-a970-19a398a270b6-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:42.575768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.575690    4453 generic.go:356] "Generic (PLEG): container finished" podID="43388545-8b63-4978-a970-19a398a270b6" containerID="cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387" exitCode=0
Oct 30 12:09:42.575768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.575743    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"
Oct 30 12:09:42.575949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.575772    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" event={"ID":"43388545-8b63-4978-a970-19a398a270b6","Type":"ContainerDied","Data":"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"}
Oct 30 12:09:42.575949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.575827    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2" event={"ID":"43388545-8b63-4978-a970-19a398a270b6","Type":"ContainerDied","Data":"5cd04074f71f486eda5db569231ec9421d686d9985206f6d667ac5410ed793da"}
Oct 30 12:09:42.575949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.575852    4453 scope.go:122] "RemoveContainer" containerID="cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"
Oct 30 12:09:42.585832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.585816    4453 scope.go:122] "RemoveContainer" containerID="cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"
Oct 30 12:09:42.586114 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:42.586094    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387\": container with ID starting with cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387 not found: ID does not exist" containerID="cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"
Oct 30 12:09:42.586168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.586122    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387"} err="failed to get container status \"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387\": rpc error: code = NotFound desc = could not find container \"cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387\": container with ID starting with cc7db5568b096151471db392a76d9612117456bb1e78ccdf56e8bd8922a5d387 not found: ID does not exist"
Oct 30 12:09:42.624256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.624213    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:09:42.637119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.637089    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-jhtz2"]
Oct 30 12:09:42.918920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:42.918537    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43388545-8b63-4978-a970-19a398a270b6" path="/var/lib/kubelet/pods/43388545-8b63-4978-a970-19a398a270b6/volumes"
Oct 30 12:09:50.169408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:50.163677    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:50.169408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:50.163960    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" containerName="istio-proxy" containerID="cri-o://9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3" gracePeriod=30
Oct 30 12:09:51.437317 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:51.437277    4453 patch_prober.go:29] interesting pod/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:09:51.437724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:51.437372    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:09:55.442672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.442649    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:55.581723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581655    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581714    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581782    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581816    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581837    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581884    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.581924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581909    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:55.582198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581932    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.582198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.581964    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.582198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582016    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:55.582198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582048    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\") pod \"78e138d4-0462-461d-b3e7-5f30668330b3\" (UID: \"78e138d4-0462-461d-b3e7-5f30668330b3\") "
Oct 30 12:09:55.582455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582278    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:55.582455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582297    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:09:55.582717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582678    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.582717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582708    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.582912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582725    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e138d4-0462-461d-b3e7-5f30668330b3-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.582912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582740    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.582912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.582806    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:55.584138 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.584107    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:55.584138 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.584107    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:09:55.584299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.584151    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "kube-api-access-7xbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:09:55.584299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.584184    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo" pod "78e138d4-0462-461d-b3e7-5f30668330b3" (UID: "78e138d4-0462-461d-b3e7-5f30668330b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:09:55.660709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.660678    4453 generic.go:356] "Generic (PLEG): container finished" podID="78e138d4-0462-461d-b3e7-5f30668330b3" containerID="9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3" exitCode=0
Oct 30 12:09:55.660846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.660720    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"
Oct 30 12:09:55.660846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.660757    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" event={"ID":"78e138d4-0462-461d-b3e7-5f30668330b3","Type":"ContainerDied","Data":"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"}
Oct 30 12:09:55.660846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.660822    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59" event={"ID":"78e138d4-0462-461d-b3e7-5f30668330b3","Type":"ContainerDied","Data":"465811bfec723709e86baea5caaff3c732510b2ac5985b547790290fa760e898"}
Oct 30 12:09:55.661018 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.660851    4453 scope.go:122] "RemoveContainer" containerID="9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"
Oct 30 12:09:55.673712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.672462    4453 scope.go:122] "RemoveContainer" containerID="9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"
Oct 30 12:09:55.673712 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:09:55.673681    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3\": container with ID starting with 9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3 not found: ID does not exist" containerID="9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"
Oct 30 12:09:55.673858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.673716    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3"} err="failed to get container status \"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3\": rpc error: code = NotFound desc = could not find container \"9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3\": container with ID starting with 9fae9a05cd32350bae29661def2aa7a74640da5d0c1cc1b218a6a184e0a04cd3 not found: ID does not exist"
Oct 30 12:09:55.684002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.683982    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.684002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.684004    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.684125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.684012    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e138d4-0462-461d-b3e7-5f30668330b3-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.684125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.684021    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbt8\" (UniqueName: \"kubernetes.io/projected/78e138d4-0462-461d-b3e7-5f30668330b3-kube-api-access-7xbt8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.684125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.684030    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e138d4-0462-461d-b3e7-5f30668330b3-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:09:55.706794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.706771    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:55.716033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:55.716009    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-afbe5f-openshift-default-666c6686c5-82b59"]
Oct 30 12:09:56.912829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:56.912798    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" path="/var/lib/kubelet/pods/78e138d4-0462-461d-b3e7-5f30668330b3/volumes"
Oct 30 12:09:59.764746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:59.764707    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 12:09:59.765203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:09:59.764925    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" containerID="cri-o://8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb" gracePeriod=30
Oct 30 12:10:03.505907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:03.505872    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-fqn85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:10:03.506317 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:03.505932    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:10:05.037769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.037706    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 12:10:05.203733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203692    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.203924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203745    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.203924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203772    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.203924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203808    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.203924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203834    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.203924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.203900    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.204204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204020    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.204204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204091    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.204204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204162    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.204204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204183    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.204441 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204220    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\") pod \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\" (UID: \"65f4eaa7-4504-49b2-a03f-bc1e830a3828\") "
Oct 30 12:10:05.204441 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204413    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.204549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204445    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.204549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204494    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:10:05.204805 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204784    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.204874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204808    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.204874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204823    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.204874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204835    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.204874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.204848    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.206226 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.206196    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:10:05.206364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.206252    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:05.206672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.206638    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "kube-api-access-wblf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:05.206780 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.206711    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo" pod "65f4eaa7-4504-49b2-a03f-bc1e830a3828" (UID: "65f4eaa7-4504-49b2-a03f-bc1e830a3828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:10:05.305884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.305809    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wblf2\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-kube-api-access-wblf2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.305884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.305839    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.305884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.305848    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.305884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.305857    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f4eaa7-4504-49b2-a03f-bc1e830a3828-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:05.748219 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.748177    4453 generic.go:356] "Generic (PLEG): container finished" podID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerID="8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb" exitCode=0
Oct 30 12:10:05.748422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.748231    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"
Oct 30 12:10:05.748422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.748257    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" event={"ID":"65f4eaa7-4504-49b2-a03f-bc1e830a3828","Type":"ContainerDied","Data":"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"}
Oct 30 12:10:05.748422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.748313    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85" event={"ID":"65f4eaa7-4504-49b2-a03f-bc1e830a3828","Type":"ContainerDied","Data":"293e7e2da1c7b2eb6e95d4bbdf59aba57a3455ef70d131cd1201d9e00828e511"}
Oct 30 12:10:05.748422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.748348    4453 scope.go:122] "RemoveContainer" containerID="8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"
Oct 30 12:10:05.758766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.758749    4453 scope.go:122] "RemoveContainer" containerID="8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"
Oct 30 12:10:05.759050 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:10:05.759034    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb\": container with ID starting with 8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb not found: ID does not exist" containerID="8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"
Oct 30 12:10:05.759093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.759058    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb"} err="failed to get container status \"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb\": rpc error: code = NotFound desc = could not find container \"8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb\": container with ID starting with 8e754bd1305f48756a2b57a49ee0be74febeacd161ecd97883dca067fdd03fbb not found: ID does not exist"
Oct 30 12:10:05.802769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.802741    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 12:10:05.811708 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:05.811668    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-fqn85"]
Oct 30 12:10:06.911843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:06.911808    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" path="/var/lib/kubelet/pods/65f4eaa7-4504-49b2-a03f-bc1e830a3828/volumes"
Oct 30 12:10:07.570112 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:10:07.570073    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:10:19.647586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.647552    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7374-755b13"]
Oct 30 12:10:19.648131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648116    4453 cpu_manager.go:414] "containerMap: removing container" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy"
Oct 30 12:10:19.648181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648131    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy"
Oct 30 12:10:19.648181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648155    4453 cpu_manager.go:414] "containerMap: removing container" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy"
Oct 30 12:10:19.648181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648160    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy"
Oct 30 12:10:19.648181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648172    4453 cpu_manager.go:414] "containerMap: removing container" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" containerName="istio-proxy"
Oct 30 12:10:19.648181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648177    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" containerName="istio-proxy"
Oct 30 12:10:19.648361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648264    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f4eaa7-4504-49b2-a03f-bc1e830a3828" containerName="istio-proxy"
Oct 30 12:10:19.648361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648273    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e138d4-0462-461d-b3e7-5f30668330b3" containerName="istio-proxy"
Oct 30 12:10:19.648361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.648282    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="43388545-8b63-4978-a970-19a398a270b6" containerName="istio-proxy"
Oct 30 12:10:19.653202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.653185    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:19.672021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.671966    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-755b13"]
Oct 30 12:10:19.676972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.676939    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 12:10:19.790482 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.790449    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8z8\" (UniqueName: \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\") pod \"test-curl-7374-755b13\" (UID: \"dd9ba6cb-c222-45c4-a861-8751afa37d6d\") " pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:19.891385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.891323    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8z8\" (UniqueName: \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\") pod \"test-curl-7374-755b13\" (UID: \"dd9ba6cb-c222-45c4-a861-8751afa37d6d\") " pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:19.911712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.911652    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8z8\" (UniqueName: \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\") pod \"test-curl-7374-755b13\" (UID: \"dd9ba6cb-c222-45c4-a861-8751afa37d6d\") " pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:19.970201 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:19.970179    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:20.269519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:20.269489    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7374-755b13"]
Oct 30 12:10:20.270829 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:10:20.270807    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd9ba6cb_c222_45c4_a861_8751afa37d6d.slice/crio-9876622d502a839548454f81bfe8514eb644c8f1d61c93664d7ee51c2a8dad31 WatchSource:0}: Error finding container 9876622d502a839548454f81bfe8514eb644c8f1d61c93664d7ee51c2a8dad31: Status 404 returned error can't find the container with id 9876622d502a839548454f81bfe8514eb644c8f1d61c93664d7ee51c2a8dad31
Oct 30 12:10:20.840793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:20.840758    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-755b13" event={"ID":"dd9ba6cb-c222-45c4-a861-8751afa37d6d","Type":"ContainerStarted","Data":"9876622d502a839548454f81bfe8514eb644c8f1d61c93664d7ee51c2a8dad31"}
Oct 30 12:10:24.978995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:24.978949    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:10:30.900906 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:30.900867    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7374-755b13" event={"ID":"dd9ba6cb-c222-45c4-a861-8751afa37d6d","Type":"ContainerStarted","Data":"629a46abbd7865712493bce1f75c838eac058f15da816c8c6c8b9d497599b2f4"}
Oct 30 12:10:30.937795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:30.937745    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7374-755b13" podStartSLOduration=1.432729116 podStartE2EDuration="11.937730097s" totalImagesPullingTime="10.505000981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:10:30.934811165 +0000 UTC m=+2430.155096074" watchObservedRunningTime="2026-10-30 12:10:30.937730097 +0000 UTC m=+2430.158015008"
Oct 30 12:10:37.569483 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:10:37.569443    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:10:41.807820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:41.807788    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7374-755b13"]
Oct 30 12:10:41.808723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:41.808679    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7374-755b13" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" containerName="test-curl-7374-755b13" containerID="cri-o://629a46abbd7865712493bce1f75c838eac058f15da816c8c6c8b9d497599b2f4" gracePeriod=2
Oct 30 12:10:41.812308 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:41.812279    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7374-755b13"]
Oct 30 12:10:41.821108 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:10:41.821077    4453 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7374-755b13\" is forbidden: User \"system:node:ip-10-0-143-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-18.ec2.internal' and this object" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:43.995106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:43.995069    4453 generic.go:356] "Generic (PLEG): container finished" podID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" containerID="629a46abbd7865712493bce1f75c838eac058f15da816c8c6c8b9d497599b2f4" exitCode=137
Oct 30 12:10:44.144212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:44.144189    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:44.255678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:44.255642    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\" (UniqueName: \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\") pod \"dd9ba6cb-c222-45c4-a861-8751afa37d6d\" (UID: \"dd9ba6cb-c222-45c4-a861-8751afa37d6d\") "
Oct 30 12:10:44.257816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:44.257790    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8" pod "dd9ba6cb-c222-45c4-a861-8751afa37d6d" (UID: "dd9ba6cb-c222-45c4-a861-8751afa37d6d"). InnerVolumeSpecName "kube-api-access-fg8z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:10:44.357644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:44.357609    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg8z8\" (UniqueName: \"kubernetes.io/projected/dd9ba6cb-c222-45c4-a861-8751afa37d6d-kube-api-access-fg8z8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:10:44.912340 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:44.912304    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" path="/var/lib/kubelet/pods/dd9ba6cb-c222-45c4-a861-8751afa37d6d/volumes"
Oct 30 12:10:45.002850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:45.002823    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7374-755b13"
Oct 30 12:10:45.003222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:10:45.002853    4453 scope.go:122] "RemoveContainer" containerID="629a46abbd7865712493bce1f75c838eac058f15da816c8c6c8b9d497599b2f4"
Oct 30 12:11:07.570849 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:11:07.570810    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:11:37.322777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:37.322701    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:11:37.323304 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:37.322898    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy" containerID="cri-o://1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a" gracePeriod=30
Oct 30 12:11:37.598629 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:11:37.598537    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:11:38.919048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:38.919019    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:11:38.919462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:38.919209    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy" containerID="cri-o://75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83" gracePeriod=30
Oct 30 12:11:42.618023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.617996    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:11:42.712970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.712937    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.712979    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713006    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713040    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713110    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713196    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713217    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:42.713663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713274    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713422    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713456    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\") pod \"a2357682-dde3-428d-87d2-3f9cc8055270\" (UID: \"a2357682-dde3-428d-87d2-3f9cc8055270\") "
Oct 30 12:11:42.713825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713735    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:42.713893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.713821    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:42.714153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714045    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:42.714270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714246    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:42.714619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714592    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.714741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714624    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2357682-dde3-428d-87d2-3f9cc8055270-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.714741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714640    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.714741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714654    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.714741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.714668    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.716293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.716257    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:42.716293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.716278    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:11:42.716480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.716382    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "kube-api-access-89m4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:42.716935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.716912    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy" pod "a2357682-dde3-428d-87d2-3f9cc8055270" (UID: "a2357682-dde3-428d-87d2-3f9cc8055270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:42.815470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.815438    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.815470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.815468    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2357682-dde3-428d-87d2-3f9cc8055270-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.815648 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.815477    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89m4p\" (UniqueName: \"kubernetes.io/projected/a2357682-dde3-428d-87d2-3f9cc8055270-kube-api-access-89m4p\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:42.815648 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:42.815487    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2357682-dde3-428d-87d2-3f9cc8055270-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:43.387772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.387738    4453 generic.go:356] "Generic (PLEG): container finished" podID="a2357682-dde3-428d-87d2-3f9cc8055270" containerID="1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a" exitCode=0
Oct 30 12:11:43.387966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.387790    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"
Oct 30 12:11:43.387966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.387813    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" event={"ID":"a2357682-dde3-428d-87d2-3f9cc8055270","Type":"ContainerDied","Data":"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"}
Oct 30 12:11:43.387966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.387871    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6" event={"ID":"a2357682-dde3-428d-87d2-3f9cc8055270","Type":"ContainerDied","Data":"9d765b038330bb18a7eb2413a38ad4c9969a224a6a80d48a48fd1e3c99109736"}
Oct 30 12:11:43.387966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.387898    4453 scope.go:122] "RemoveContainer" containerID="1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"
Oct 30 12:11:43.402407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.402387    4453 scope.go:122] "RemoveContainer" containerID="1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"
Oct 30 12:11:43.402708 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:11:43.402688    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a\": container with ID starting with 1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a not found: ID does not exist" containerID="1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"
Oct 30 12:11:43.402754 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.402719    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a"} err="failed to get container status \"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a\": rpc error: code = NotFound desc = could not find container \"1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a\": container with ID starting with 1d7c3afeca1e565231151b7bacdd6df6e49702645eca463efeccb8706201541a not found: ID does not exist"
Oct 30 12:11:43.443840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.443774    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:11:43.452878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:43.452850    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-mzfw6"]
Oct 30 12:11:44.207160 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.207094    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:11:44.333436 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333406    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333500    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333559    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333583    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333635    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333654    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333657    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:44.333904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333752    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.333904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333794    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:44.333904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333820    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.334064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333905    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\") pod \"63607f10-f85d-449c-ba61-f3c20e520d52\" (UID: \"63607f10-f85d-449c-ba61-f3c20e520d52\") "
Oct 30 12:11:44.334064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.333979    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:44.334184 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334114    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:44.334184 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334132    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:44.334562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334536    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.334660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334564    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63607f10-f85d-449c-ba61-f3c20e520d52-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.334660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334578    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.334660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334590    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.334660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.334603    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.335969 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.335945    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:44.336056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.335975    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:44.336056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.335981    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:11:44.336056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.336041    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5" pod "63607f10-f85d-449c-ba61-f3c20e520d52" (UID: "63607f10-f85d-449c-ba61-f3c20e520d52"). InnerVolumeSpecName "kube-api-access-x6xd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:44.396683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.396650    4453 generic.go:356] "Generic (PLEG): container finished" podID="63607f10-f85d-449c-ba61-f3c20e520d52" containerID="75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83" exitCode=0
Oct 30 12:11:44.396844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.396705    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"
Oct 30 12:11:44.396844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.396734    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" event={"ID":"63607f10-f85d-449c-ba61-f3c20e520d52","Type":"ContainerDied","Data":"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"}
Oct 30 12:11:44.396844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.396791    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd" event={"ID":"63607f10-f85d-449c-ba61-f3c20e520d52","Type":"ContainerDied","Data":"ed77936b2eb5cb4ed4dea8dcf57470e8fa91641425a5974859d472b14dfa2649"}
Oct 30 12:11:44.396844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.396818    4453 scope.go:122] "RemoveContainer" containerID="75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"
Oct 30 12:11:44.408998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.408977    4453 scope.go:122] "RemoveContainer" containerID="75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"
Oct 30 12:11:44.409312 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:11:44.409286    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83\": container with ID starting with 75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83 not found: ID does not exist" containerID="75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"
Oct 30 12:11:44.409418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.409322    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83"} err="failed to get container status \"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83\": rpc error: code = NotFound desc = could not find container \"75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83\": container with ID starting with 75d80811e9b09d809cafd13513c495959296c1fb57357dd5bfb917b59890cd83 not found: ID does not exist"
Oct 30 12:11:44.435880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.435855    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63607f10-f85d-449c-ba61-f3c20e520d52-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.435973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.435884    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63607f10-f85d-449c-ba61-f3c20e520d52-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.435973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.435898    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.435973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.435910    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xd5\" (UniqueName: \"kubernetes.io/projected/63607f10-f85d-449c-ba61-f3c20e520d52-kube-api-access-x6xd5\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:44.441877 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.441850    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:11:44.446063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.446040    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-7zphd"]
Oct 30 12:11:44.912455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.912404    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" path="/var/lib/kubelet/pods/63607f10-f85d-449c-ba61-f3c20e520d52/volumes"
Oct 30 12:11:44.912970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:44.912953    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" path="/var/lib/kubelet/pods/a2357682-dde3-428d-87d2-3f9cc8055270/volumes"
Oct 30 12:11:49.814470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:49.814435    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 12:11:49.814919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:49.814656    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" containerID="cri-o://052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770" gracePeriod=30
Oct 30 12:11:54.148776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:54.148735    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:11:54.149226 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:54.148790    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:11:55.082404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.082342    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 12:11:55.183073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183037    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183096    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183154    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183238    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183274    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183323    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183355    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183407    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:55.183536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183452    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183544    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183618    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\") pod \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\" (UID: \"38f7574c-e999-4d5b-b6f1-68b5126dfff4\") "
Oct 30 12:11:55.183994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.183828    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:55.184160 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.184137    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:11:55.184218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.184196    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.184287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.184220    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.184287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.184237    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.184287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.184194    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:55.185804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.185779    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "kube-api-access-b4s5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:55.186042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.186017    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:11:55.186042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.186034    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:11:55.186192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.186174    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo" pod "38f7574c-e999-4d5b-b6f1-68b5126dfff4" (UID: "38f7574c-e999-4d5b-b6f1-68b5126dfff4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:11:55.285231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285197    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.285231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285224    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.285231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285232    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.285231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285242    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.285515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285250    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4s5k\" (UniqueName: \"kubernetes.io/projected/38f7574c-e999-4d5b-b6f1-68b5126dfff4-kube-api-access-b4s5k\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.285515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.285259    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38f7574c-e999-4d5b-b6f1-68b5126dfff4-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:11:55.466859 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.466827    4453 generic.go:356] "Generic (PLEG): container finished" podID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerID="052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770" exitCode=0
Oct 30 12:11:55.467041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.466877    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"
Oct 30 12:11:55.467041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.466919    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" event={"ID":"38f7574c-e999-4d5b-b6f1-68b5126dfff4","Type":"ContainerDied","Data":"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"}
Oct 30 12:11:55.467041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.466980    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg" event={"ID":"38f7574c-e999-4d5b-b6f1-68b5126dfff4","Type":"ContainerDied","Data":"9d8dd8086cd1d44815536ba067b0ed166645376edc500249698c2ef00a05a8b0"}
Oct 30 12:11:55.467041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.467005    4453 scope.go:122] "RemoveContainer" containerID="052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"
Oct 30 12:11:55.477077 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.477059    4453 scope.go:122] "RemoveContainer" containerID="052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"
Oct 30 12:11:55.477370 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:11:55.477351    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770\": container with ID starting with 052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770 not found: ID does not exist" containerID="052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"
Oct 30 12:11:55.477440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.477378    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770"} err="failed to get container status \"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770\": rpc error: code = NotFound desc = could not find container \"052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770\": container with ID starting with 052cf3bb722addd3e3406b6e1d0020af6a50f2e9b9b3590f5196a0585642e770 not found: ID does not exist"
Oct 30 12:11:55.511625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.511601    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 12:11:55.515371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:55.515350    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-dh8bg"]
Oct 30 12:11:56.911796 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:11:56.911763    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" path="/var/lib/kubelet/pods/38f7574c-e999-4d5b-b6f1-68b5126dfff4/volumes"
Oct 30 12:12:01.953903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:01.953749    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:12:03.925791 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.925755    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:12:03.926295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926280    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy"
Oct 30 12:12:03.926359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926296    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy"
Oct 30 12:12:03.926359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926316    4453 cpu_manager.go:414] "containerMap: removing container" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy"
Oct 30 12:12:03.926359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926321    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy"
Oct 30 12:12:03.926359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926350    4453 cpu_manager.go:414] "containerMap: removing container" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy"
Oct 30 12:12:03.926359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926356    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926369    4453 cpu_manager.go:414] "containerMap: removing container" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" containerName="test-curl-7374-755b13"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926374    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" containerName="test-curl-7374-755b13"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926450    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="63607f10-f85d-449c-ba61-f3c20e520d52" containerName="istio-proxy"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926459    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f7574c-e999-4d5b-b6f1-68b5126dfff4" containerName="istio-proxy"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926467    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9ba6cb-c222-45c4-a861-8751afa37d6d" containerName="test-curl-7374-755b13"
Oct 30 12:12:03.926532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.926476    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2357682-dde3-428d-87d2-3f9cc8055270" containerName="istio-proxy"
Oct 30 12:12:03.931066 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.931049    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:03.952561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.952526    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-737f64\""
Oct 30 12:12:03.952561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.952531    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-737f64-dockercfg-6gv5d\""
Oct 30 12:12:03.955641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:03.955607    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:12:04.056311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.056147    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:12:04.063235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.063207    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.114390 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.114355    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:12:04.115072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.115040    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twlvr\" (UniqueName: \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.115264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.115103    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.216937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.216861    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rbq\" (UniqueName: \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.217114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.217062    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twlvr\" (UniqueName: \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.217114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.217099    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.217228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.217135    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.218417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.218386    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.238095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.238074    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twlvr\" (UniqueName: \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\") pod \"payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.245283 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.245262    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:04.320260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.319188    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.320260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.319307    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rbq\" (UniqueName: \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.320550 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.320496    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.340779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.340747    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rbq\" (UniqueName: \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\") pod \"payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.381401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.381369    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:04.547870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.547843    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:12:04.549259 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:04.549227    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54768444_c294_46d6_bd60_5b2b8c0d6157.slice/crio-d19de1a62ae22aea82afd38999790a7379d07a9c7fcc53cc1cde42fef41f153e WatchSource:0}: Error finding container d19de1a62ae22aea82afd38999790a7379d07a9c7fcc53cc1cde42fef41f153e: Status 404 returned error can't find the container with id d19de1a62ae22aea82afd38999790a7379d07a9c7fcc53cc1cde42fef41f153e
Oct 30 12:12:04.687145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:04.687119    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:12:04.688244 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:04.688210    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59684744_248f_41fb_833b_e92effd276d0.slice/crio-a8f60a3ffa0ae48569a44596a9acadc55fa61acb9c231f96936968ccdec67e60 WatchSource:0}: Error finding container a8f60a3ffa0ae48569a44596a9acadc55fa61acb9c231f96936968ccdec67e60: Status 404 returned error can't find the container with id a8f60a3ffa0ae48569a44596a9acadc55fa61acb9c231f96936968ccdec67e60
Oct 30 12:12:05.533553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.533511    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" event={"ID":"54768444-c294-46d6-bd60-5b2b8c0d6157","Type":"ContainerStarted","Data":"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"}
Oct 30 12:12:05.534002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.533582    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:05.534002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.533605    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" event={"ID":"54768444-c294-46d6-bd60-5b2b8c0d6157","Type":"ContainerStarted","Data":"d19de1a62ae22aea82afd38999790a7379d07a9c7fcc53cc1cde42fef41f153e"}
Oct 30 12:12:05.535170 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.535145    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" event={"ID":"59684744-248f-41fb-833b-e92effd276d0","Type":"ContainerStarted","Data":"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"}
Oct 30 12:12:05.535267 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.535183    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" event={"ID":"59684744-248f-41fb-833b-e92effd276d0","Type":"ContainerStarted","Data":"a8f60a3ffa0ae48569a44596a9acadc55fa61acb9c231f96936968ccdec67e60"}
Oct 30 12:12:05.535267 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.535203    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:05.570351 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.570288    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" podStartSLOduration=2.570276833 podStartE2EDuration="2.570276833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:05.568165285 +0000 UTC m=+2524.788450195" watchObservedRunningTime="2026-10-30 12:12:05.570276833 +0000 UTC m=+2524.790561743"
Oct 30 12:12:05.604276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:05.604225    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" podStartSLOduration=2.604212562 podStartE2EDuration="2.604212562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:05.602444528 +0000 UTC m=+2524.822729438" watchObservedRunningTime="2026-10-30 12:12:05.604212562 +0000 UTC m=+2524.824497473"
Oct 30 12:12:07.569421 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:12:07.569387    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:12:10.099294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.099261    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:12:10.103476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.103458    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.115967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.115945    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2bc4e2\""
Oct 30 12:12:10.125362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.125321    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2bc4e2-dockercfg-sxwhd\""
Oct 30 12:12:10.128664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.128642    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:12:10.295005 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.294969    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.295235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.295213    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvrs\" (UniqueName: \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.400391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.396890    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.400391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.397011    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvrs\" (UniqueName: \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.400391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.398020    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.417842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.417816    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvrs\" (UniqueName: \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\") pod \"payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.418033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.418019    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:10.717447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:10.717354    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:12:10.723252 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:10.723224    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8e5aad_5b8b_40b3_b073_fcb9d09ebcd9.slice/crio-06b8ee30e524765e344ed8a441a96c7638a3fa4a3a806a809ad0591ce061481a WatchSource:0}: Error finding container 06b8ee30e524765e344ed8a441a96c7638a3fa4a3a806a809ad0591ce061481a: Status 404 returned error can't find the container with id 06b8ee30e524765e344ed8a441a96c7638a3fa4a3a806a809ad0591ce061481a
Oct 30 12:12:11.569868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:11.569828    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" event={"ID":"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9","Type":"ContainerStarted","Data":"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"}
Oct 30 12:12:11.570291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:11.569899    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" event={"ID":"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9","Type":"ContainerStarted","Data":"06b8ee30e524765e344ed8a441a96c7638a3fa4a3a806a809ad0591ce061481a"}
Oct 30 12:12:11.570291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:11.570040    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:11.603957 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:11.603905    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" podStartSLOduration=1.603893518 podStartE2EDuration="1.603893518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:11.603238349 +0000 UTC m=+2530.823523258" watchObservedRunningTime="2026-10-30 12:12:11.603893518 +0000 UTC m=+2530.824178484"
Oct 30 12:12:16.544070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:16.544036    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:12:16.544453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:16.544123    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:12:16.975280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:16.975241    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" podUID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" containerName="main" containerID="cri-o://f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9" gracePeriod=45
Oct 30 12:12:17.247041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.247016    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:12:17.278865 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.278840    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.278885    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\" (UniqueName: \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.278936    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.278967    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279005    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279076    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\") pod \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\" (UID: \"ac10d7e4-474c-4c44-92e0-55d04eeeb31e\") "
Oct 30 12:12:17.279209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279123    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:17.279350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279227    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:17.279350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279242    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:17.279754 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279731    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.279893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279758    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.279893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.279773    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.281324 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.281296    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:17.281460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.281365    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:12:17.281460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.281395    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd" pod "ac10d7e4-474c-4c44-92e0-55d04eeeb31e" (UID: "ac10d7e4-474c-4c44-92e0-55d04eeeb31e"). InnerVolumeSpecName "kube-api-access-d8ltd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:17.380518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.380482    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ltd\" (UniqueName: \"kubernetes.io/projected/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-kube-api-access-d8ltd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.380518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.380510    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.380518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.380523    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac10d7e4-474c-4c44-92e0-55d04eeeb31e-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:17.604904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.604804    4453 generic.go:356] "Generic (PLEG): container finished" podID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" containerID="f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9" exitCode=0
Oct 30 12:12:17.604904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.604857    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" event={"ID":"ac10d7e4-474c-4c44-92e0-55d04eeeb31e","Type":"ContainerDied","Data":"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"}
Oct 30 12:12:17.604904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.604892    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt" event={"ID":"ac10d7e4-474c-4c44-92e0-55d04eeeb31e","Type":"ContainerDied","Data":"ebd3f5d367549e650ecd8f0ba5d49d164a4971d4fa591082f9ffea889bed5cbf"}
Oct 30 12:12:17.605506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.604910    4453 scope.go:122] "RemoveContainer" containerID="f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"
Oct 30 12:12:17.605506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.604861    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"
Oct 30 12:12:17.615323 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.615303    4453 scope.go:122] "RemoveContainer" containerID="f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"
Oct 30 12:12:17.615676 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:12:17.615657    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9\": container with ID starting with f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9 not found: ID does not exist" containerID="f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"
Oct 30 12:12:17.615725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.615685    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9"} err="failed to get container status \"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9\": rpc error: code = NotFound desc = could not find container \"f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9\": container with ID starting with f8e4e855f11638f3f131c55b721af63ab26007d315fa778447af1b4fb563b6a9 not found: ID does not exist"
Oct 30 12:12:17.648655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.648620    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:12:17.657480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:17.657450    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-87f2e6/auth-test-model-87f2e6-kserve-76bb657c89-dltdt"]
Oct 30 12:12:18.224113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:18.222730    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:12:18.911764 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:18.911721    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" path="/var/lib/kubelet/pods/ac10d7e4-474c-4c44-92e0-55d04eeeb31e/volumes"
Oct 30 12:12:19.374389 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.374356    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:12:19.375246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.375229    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" containerName="main"
Oct 30 12:12:19.375319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.375248    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" containerName="main"
Oct 30 12:12:19.375414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.375395    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac10d7e4-474c-4c44-92e0-55d04eeeb31e" containerName="main"
Oct 30 12:12:19.380968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.380944    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.393506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.393484    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3e23ec-openshift-default-dockercfg-fgff5\""
Oct 30 12:12:19.408992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.408962    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:12:19.409935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.409721    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.409935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.409796    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhxw\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.409935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.409832    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.409935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.409897    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.410194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.409973    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.410194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.410031    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.410194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.410056    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.410194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.410090    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.410194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.410127    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511090    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511180    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511213    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511247    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511285    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511398    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511465    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhxw\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511507    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.511572    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.519354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.512959    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.520488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.520460    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.527352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.522954    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.527352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.523189    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.549199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.547111    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.555062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.555021    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhxw\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.559449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.559404    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.565773 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.564869    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.592841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.592803    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\") pod \"e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") " pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:19.700561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:19.700483    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:20.024683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.024652    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:12:20.025740 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:20.025717    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d61d5ef_9e3f_49b3_8049_a6bb8fe4e69c.slice/crio-d73bfdc705cfa45f0aec9519f35a06d895f7e10c7c4433fa383b0fcab1a9c8a6 WatchSource:0}: Error finding container d73bfdc705cfa45f0aec9519f35a06d895f7e10c7c4433fa383b0fcab1a9c8a6: Status 404 returned error can't find the container with id d73bfdc705cfa45f0aec9519f35a06d895f7e10c7c4433fa383b0fcab1a9c8a6
Oct 30 12:12:20.028273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.028238    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:12:20.028378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.028305    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:12:20.028378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.028351    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:12:20.625998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.625961    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerStarted","Data":"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"}
Oct 30 12:12:20.626169 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.626016    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerStarted","Data":"d73bfdc705cfa45f0aec9519f35a06d895f7e10c7c4433fa383b0fcab1a9c8a6"}
Oct 30 12:12:20.677225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.677163    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podStartSLOduration=1.677146369 podStartE2EDuration="1.677146369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:20.675256581 +0000 UTC m=+2539.895541492" watchObservedRunningTime="2026-10-30 12:12:20.677146369 +0000 UTC m=+2539.897431282"
Oct 30 12:12:20.701245 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:20.701204    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:21.724367 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:21.723636    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:22.578729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:22.578697    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:12:22.672503 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:22.672470    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:22.673569 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:22.673542    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:12:29.904088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:29.904051    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:12:29.908279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:29.908258    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:29.920620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:29.920592    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-737f64-serving-cert\""
Oct 30 12:12:29.933523 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:29.933496    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:12:30.045162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.045126    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.045344 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.045279    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9qf\" (UniqueName: \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.045422 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.045402    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.045482 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.045444    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.148736 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.148679    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9qf\" (UniqueName: \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.149075 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.149047    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.149246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.149225    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.149549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.149522    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.151876 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.151846    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.155212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.153666    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.157784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.157760    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.175536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.175510    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9qf\" (UniqueName: \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\") pod \"maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.222177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.222146    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:30.523036 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.523004    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:12:30.524204 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:30.524178    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b15dee6_bff0_45cb_91f8_73cf81cccdd6.slice/crio-c17740864f907eb7949109de8050e10c305054ac832edceaeef88c250e0f2503 WatchSource:0}: Error finding container c17740864f907eb7949109de8050e10c305054ac832edceaeef88c250e0f2503: Status 404 returned error can't find the container with id c17740864f907eb7949109de8050e10c305054ac832edceaeef88c250e0f2503
Oct 30 12:12:30.525669 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.525649    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:12:30.718577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:30.718532    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" event={"ID":"9b15dee6-bff0-45cb-91f8-73cf81cccdd6","Type":"ContainerStarted","Data":"c17740864f907eb7949109de8050e10c305054ac832edceaeef88c250e0f2503"}
Oct 30 12:12:31.726062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:31.726013    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" event={"ID":"9b15dee6-bff0-45cb-91f8-73cf81cccdd6","Type":"ContainerStarted","Data":"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"}
Oct 30 12:12:31.726485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:31.726114    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:31.765783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:31.765734    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" podStartSLOduration=2.415041931 podStartE2EDuration="2.765719949s" totalImagesPullingTime="350.678018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:31.762767268 +0000 UTC m=+2550.983052177" watchObservedRunningTime="2026-10-30 12:12:31.765719949 +0000 UTC m=+2550.986004859"
Oct 30 12:12:33.244572 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.244530    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" podUID="eb91fd83-4956-458b-8805-9153ef072e25" containerName="main" containerID="cri-o://10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987" gracePeriod=45
Oct 30 12:12:33.515484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.515460    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:12:33.606430 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606396    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606585 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606485    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606585 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606517    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\" (UniqueName: \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606585 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606542    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606592    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606613    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\") pod \"eb91fd83-4956-458b-8805-9153ef072e25\" (UID: \"eb91fd83-4956-458b-8805-9153ef072e25\") "
Oct 30 12:12:33.606919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606891    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:33.606919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.606903    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:33.607211 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.607187    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:33.607326 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.607311    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.607408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.607347    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.607408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.607362    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.609354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.609315    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:33.609427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.609388    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:12:33.609427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.609394    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6" pod "eb91fd83-4956-458b-8805-9153ef072e25" (UID: "eb91fd83-4956-458b-8805-9153ef072e25"). InnerVolumeSpecName "kube-api-access-h28w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:33.708316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.708286    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb91fd83-4956-458b-8805-9153ef072e25-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.708316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.708311    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb91fd83-4956-458b-8805-9153ef072e25-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.708485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.708324    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28w6\" (UniqueName: \"kubernetes.io/projected/eb91fd83-4956-458b-8805-9153ef072e25-kube-api-access-h28w6\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:33.739494 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.739462    4453 generic.go:356] "Generic (PLEG): container finished" podID="eb91fd83-4956-458b-8805-9153ef072e25" containerID="10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987" exitCode=0
Oct 30 12:12:33.739611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.739509    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"
Oct 30 12:12:33.739611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.739538    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" event={"ID":"eb91fd83-4956-458b-8805-9153ef072e25","Type":"ContainerDied","Data":"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"}
Oct 30 12:12:33.739611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.739578    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv" event={"ID":"eb91fd83-4956-458b-8805-9153ef072e25","Type":"ContainerDied","Data":"aba04dedce90c516af475a9af7b2fa8c9d7000f14a6931d564d8cba103197563"}
Oct 30 12:12:33.739611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.739596    4453 scope.go:122] "RemoveContainer" containerID="10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"
Oct 30 12:12:33.749838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.749818    4453 scope.go:122] "RemoveContainer" containerID="10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"
Oct 30 12:12:33.750102 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:12:33.750084    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987\": container with ID starting with 10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987 not found: ID does not exist" containerID="10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"
Oct 30 12:12:33.750143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.750112    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987"} err="failed to get container status \"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987\": rpc error: code = NotFound desc = could not find container \"10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987\": container with ID starting with 10ce2981c39219a069877bb2a6102c4edb8eeb455fc90441b30a5afd5d88a987 not found: ID does not exist"
Oct 30 12:12:33.783953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.783930    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:12:33.790080 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:33.790061    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-4a92de/auth-test-model-4a92de-kserve-77bf54676f-qd6fv"]
Oct 30 12:12:34.912669 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:34.912633    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb91fd83-4956-458b-8805-9153ef072e25" path="/var/lib/kubelet/pods/eb91fd83-4956-458b-8805-9153ef072e25/volumes"
Oct 30 12:12:34.949745 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:34.949713    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 12:12:34.949936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:34.949898    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" containerID="cri-o://705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0" gracePeriod=30
Oct 30 12:12:35.015927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.015898    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:12:35.016515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.016498    4453 cpu_manager.go:414] "containerMap: removing container" podUID="eb91fd83-4956-458b-8805-9153ef072e25" containerName="main"
Oct 30 12:12:35.016515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.016515    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb91fd83-4956-458b-8805-9153ef072e25" containerName="main"
Oct 30 12:12:35.016646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.016623    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb91fd83-4956-458b-8805-9153ef072e25" containerName="main"
Oct 30 12:12:35.022915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.022894    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.037645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.037622    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3e23ec\""
Oct 30 12:12:35.037645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.037639    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3e23ec-dockercfg-fcj6x\""
Oct 30 12:12:35.040896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.040873    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:12:35.155136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.155094    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.156462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.156432    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hxs\" (UniqueName: \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.259590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.259498    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.259590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.259546    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hxs\" (UniqueName: \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.260317 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.260295    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.280192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.280168    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hxs\" (UniqueName: \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\") pod \"payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.337425 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.337403    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.637735 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.637706    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:12:35.638535 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:35.638501    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ac29bc_b19f_4ba4_a56e_c5e7776d0860.slice/crio-35fdb76bf436f86edb4afe1b3342b460267f215467217890a275058b4285d23b WatchSource:0}: Error finding container 35fdb76bf436f86edb4afe1b3342b460267f215467217890a275058b4285d23b: Status 404 returned error can't find the container with id 35fdb76bf436f86edb4afe1b3342b460267f215467217890a275058b4285d23b
Oct 30 12:12:35.754643 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.754613    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" event={"ID":"86ac29bc-b19f-4ba4-a56e-c5e7776d0860","Type":"ContainerStarted","Data":"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"}
Oct 30 12:12:35.754779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.754666    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" event={"ID":"86ac29bc-b19f-4ba4-a56e-c5e7776d0860","Type":"ContainerStarted","Data":"35fdb76bf436f86edb4afe1b3342b460267f215467217890a275058b4285d23b"}
Oct 30 12:12:35.754779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.754716    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:35.790839 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:35.790795    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" podStartSLOduration=1.79078358 podStartE2EDuration="1.79078358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:35.789530677 +0000 UTC m=+2555.009815587" watchObservedRunningTime="2026-10-30 12:12:35.79078358 +0000 UTC m=+2555.011068491"
Oct 30 12:12:37.570527 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:12:37.570490    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:12:37.737528 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:37.737499    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:12:39.270936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:39.270895    4453 patch_prober.go:29] interesting pod/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:12:39.271378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:39.270954    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:12:40.266071 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.266043    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 12:12:40.327615 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327577    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327670    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327785    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327867    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327970    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.327969    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328033    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328068    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328098    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328179    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\") pod \"80dc15c7-58bb-475f-9f1c-11b0fa049309\" (UID: \"80dc15c7-58bb-475f-9f1c-11b0fa049309\") "
Oct 30 12:12:40.328798 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328773    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:40.328907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.328797    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.329376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.329033    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:40.329376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.329094    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:40.329376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.329254    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:12:40.331292 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.331271    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:40.331427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.331289    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:12:40.331427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.331408    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:12:40.334981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.334957    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk" pod "80dc15c7-58bb-475f-9f1c-11b0fa049309" (UID: "80dc15c7-58bb-475f-9f1c-11b0fa049309"). InnerVolumeSpecName "kube-api-access-vvjjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:12:40.430799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430761    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjjk\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-kube-api-access-vvjjk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.430799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430797    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.430799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430807    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.431134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430816    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.431134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430825    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dc15c7-58bb-475f-9f1c-11b0fa049309-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.431134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430833    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.431134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430840    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dc15c7-58bb-475f-9f1c-11b0fa049309-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.431134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.430848    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dc15c7-58bb-475f-9f1c-11b0fa049309-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:12:40.786535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.786494    4453 generic.go:356] "Generic (PLEG): container finished" podID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerID="705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0" exitCode=0
Oct 30 12:12:40.786733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.786548    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"
Oct 30 12:12:40.786733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.786574    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" event={"ID":"80dc15c7-58bb-475f-9f1c-11b0fa049309","Type":"ContainerDied","Data":"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"}
Oct 30 12:12:40.786733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.786620    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx" event={"ID":"80dc15c7-58bb-475f-9f1c-11b0fa049309","Type":"ContainerDied","Data":"7f80bf9291f1255dd79fdf4a30eb246aa2bfaa44f1413aa3f60f033547961445"}
Oct 30 12:12:40.786733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.786644    4453 scope.go:122] "RemoveContainer" containerID="705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"
Oct 30 12:12:40.797646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.797626    4453 scope.go:122] "RemoveContainer" containerID="705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"
Oct 30 12:12:40.797968 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:12:40.797949    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0\": container with ID starting with 705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0 not found: ID does not exist" containerID="705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"
Oct 30 12:12:40.798033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.797976    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0"} err="failed to get container status \"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0\": rpc error: code = NotFound desc = could not find container \"705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0\": container with ID starting with 705f85173e283890cdaa8ea7cace0f2d398db187fe11e01d7accdaae303f9ed0 not found: ID does not exist"
Oct 30 12:12:40.841037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.841009    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 12:12:40.850164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.850133    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-hn7bx"]
Oct 30 12:12:40.911964 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:40.911927    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" path="/var/lib/kubelet/pods/80dc15c7-58bb-475f-9f1c-11b0fa049309/volumes"
Oct 30 12:12:46.764224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:46.764077    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:12:53.349132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.349095    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:12:53.350062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.350039    4453 cpu_manager.go:414] "containerMap: removing container" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy"
Oct 30 12:12:53.350121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.350066    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy"
Oct 30 12:12:53.350217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.350203    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="80dc15c7-58bb-475f-9f1c-11b0fa049309" containerName="istio-proxy"
Oct 30 12:12:53.355454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.355421    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.380173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.380143    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:12:53.381089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.381050    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-846099\""
Oct 30 12:12:53.381797 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.381772    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-846099-dockercfg-btkww\""
Oct 30 12:12:53.470351 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.464917    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:12:53.476406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.475672    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.493720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.489163    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:12:53.536601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.536212    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.536601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.536290    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2774q\" (UniqueName: \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.536601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.536401    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsptx\" (UniqueName: \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.536601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.536504    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.637614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.637578    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.637804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.637629    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2774q\" (UniqueName: \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.637804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.637684    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsptx\" (UniqueName: \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.637804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.637740    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.638577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.638545    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.638577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.638566    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.662123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.662093    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2774q\" (UniqueName: \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\") pod \"payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.663022 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.662996    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsptx\" (UniqueName: \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\") pod \"payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.668718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.668687    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:53.828951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.828918    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:53.968090 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:53.968052    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:12:53.969432 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:53.969409    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada18417_a16b_4041_8a0c_cf2746c19f6a.slice/crio-6cd5de1ac2d861c40eb90397a5e93ce909d18656b3727e39b8a972c05130fab9 WatchSource:0}: Error finding container 6cd5de1ac2d861c40eb90397a5e93ce909d18656b3727e39b8a972c05130fab9: Status 404 returned error can't find the container with id 6cd5de1ac2d861c40eb90397a5e93ce909d18656b3727e39b8a972c05130fab9
Oct 30 12:12:54.127620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.127589    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:12:54.128672 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:12:54.128651    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91372140_3b06_4754_b790_b413ead12b0c.slice/crio-f6ec413aaaf15c8f6738d846223370655a2236a5797649f7e18bfca93105b35d WatchSource:0}: Error finding container f6ec413aaaf15c8f6738d846223370655a2236a5797649f7e18bfca93105b35d: Status 404 returned error can't find the container with id f6ec413aaaf15c8f6738d846223370655a2236a5797649f7e18bfca93105b35d
Oct 30 12:12:54.876556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.876519    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" event={"ID":"ada18417-a16b-4041-8a0c-cf2746c19f6a","Type":"ContainerStarted","Data":"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"}
Oct 30 12:12:54.876996 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.876588    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:12:54.876996 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.876615    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" event={"ID":"ada18417-a16b-4041-8a0c-cf2746c19f6a","Type":"ContainerStarted","Data":"6cd5de1ac2d861c40eb90397a5e93ce909d18656b3727e39b8a972c05130fab9"}
Oct 30 12:12:54.878265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.878235    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" event={"ID":"91372140-3b06-4754-b790-b413ead12b0c","Type":"ContainerStarted","Data":"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"}
Oct 30 12:12:54.878412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.878286    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" event={"ID":"91372140-3b06-4754-b790-b413ead12b0c","Type":"ContainerStarted","Data":"f6ec413aaaf15c8f6738d846223370655a2236a5797649f7e18bfca93105b35d"}
Oct 30 12:12:54.878412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.878316    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:12:54.913054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.913004    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" podStartSLOduration=1.912987872 podStartE2EDuration="1.912987872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:54.911051043 +0000 UTC m=+2574.131335953" watchObservedRunningTime="2026-10-30 12:12:54.912987872 +0000 UTC m=+2574.133272784"
Oct 30 12:12:54.947270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:12:54.947223    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" podStartSLOduration=1.947210194 podStartE2EDuration="1.947210194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:12:54.946120224 +0000 UTC m=+2574.166405137" watchObservedRunningTime="2026-10-30 12:12:54.947210194 +0000 UTC m=+2574.167495104"
Oct 30 12:13:05.888839 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:05.888810    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:13:05.889295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:05.888879    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:13:07.571043 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:13:07.571007    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:13:25.907425 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.907390    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:13:25.913255 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.913231    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:25.935552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.935524    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-846099-serving-cert\""
Oct 30 12:13:25.950712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.950663    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:13:25.954769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.953588    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:25.954769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.953666    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzn9t\" (UniqueName: \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:25.954769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.953856    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:25.954769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:25.953949    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.055403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.055369    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.055586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.055419    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzn9t\" (UniqueName: \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.055586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.055498    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.055586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.055550    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.056202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.056177    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.057840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.057811    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.057979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.057956    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.076047 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.076021    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzn9t\" (UniqueName: \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\") pod \"maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.227027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.226955    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:26.531841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:26.531810    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:13:26.532940 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:13:26.532914    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf758a8d_c9e6_42f0_a655_99fe55aaa01e.slice/crio-8b5a3637e1ef26d82f81343d2e7713da21aabdeff6f9e91c127ef97ee3f036f7 WatchSource:0}: Error finding container 8b5a3637e1ef26d82f81343d2e7713da21aabdeff6f9e91c127ef97ee3f036f7: Status 404 returned error can't find the container with id 8b5a3637e1ef26d82f81343d2e7713da21aabdeff6f9e91c127ef97ee3f036f7
Oct 30 12:13:27.117319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:27.117282    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" event={"ID":"cf758a8d-c9e6-42f0-a655-99fe55aaa01e","Type":"ContainerStarted","Data":"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"}
Oct 30 12:13:27.117783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:27.117529    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:27.117783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:27.117567    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" event={"ID":"cf758a8d-c9e6-42f0-a655-99fe55aaa01e","Type":"ContainerStarted","Data":"8b5a3637e1ef26d82f81343d2e7713da21aabdeff6f9e91c127ef97ee3f036f7"}
Oct 30 12:13:27.159352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:27.156870    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" podStartSLOduration=1.7152468 podStartE2EDuration="2.156848975s" totalImagesPullingTime="441.602175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:27.155023449 +0000 UTC m=+2606.375308362" watchObservedRunningTime="2026-10-30 12:13:27.156848975 +0000 UTC m=+2606.377133885"
Oct 30 12:13:33.130217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:33.130187    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:13:36.054352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.046377    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:13:36.066530 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.066500    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.101412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.099673    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:13:36.117766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.117718    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7b8a9d\""
Oct 30 12:13:36.118023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.117927    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7b8a9d-dockercfg-qb2vv\""
Oct 30 12:13:36.190320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.190283    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hksh\" (UniqueName: \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.190549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.190527    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.291648 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.291611    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hksh\" (UniqueName: \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.291836 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.291754    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.292458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.292436    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.312467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.312396    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hksh\" (UniqueName: \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\") pod \"payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.451083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.451038    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:36.764627 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:36.764597    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:13:36.766014 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:13:36.765981    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613a10e5_5f00_45d0_bc5f_c9a575e90a50.slice/crio-a5b1a3f4973fe3e252483c909a95a26551a94ec9a2148eaa295ec2f255ca583f WatchSource:0}: Error finding container a5b1a3f4973fe3e252483c909a95a26551a94ec9a2148eaa295ec2f255ca583f: Status 404 returned error can't find the container with id a5b1a3f4973fe3e252483c909a95a26551a94ec9a2148eaa295ec2f255ca583f
Oct 30 12:13:37.180378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:37.179670    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" event={"ID":"613a10e5-5f00-45d0-bc5f-c9a575e90a50","Type":"ContainerStarted","Data":"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"}
Oct 30 12:13:37.180378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:37.179734    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" event={"ID":"613a10e5-5f00-45d0-bc5f-c9a575e90a50","Type":"ContainerStarted","Data":"a5b1a3f4973fe3e252483c909a95a26551a94ec9a2148eaa295ec2f255ca583f"}
Oct 30 12:13:37.180378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:37.179853    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:37.217313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:37.217250    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" podStartSLOduration=1.2172379150000001 podStartE2EDuration="1.217237915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:13:37.216309874 +0000 UTC m=+2616.436594784" watchObservedRunningTime="2026-10-30 12:13:37.217237915 +0000 UTC m=+2616.437522824"
Oct 30 12:13:37.574468 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:13:37.574393    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:13:47.268536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.268494    4453 generic.go:356] "Generic (PLEG): container finished" podID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerID="5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe" exitCode=0
Oct 30 12:13:47.268536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.268526    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerDied","Data":"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"}
Oct 30 12:13:47.269155 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.269051    4453 scope.go:122] "RemoveContainer" containerID="5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"
Oct 30 12:13:47.270988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.270959    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:13:47.271070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.271029    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:13:47.271070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:47.271064    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:13:48.206354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:48.195123    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:13:48.283374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:48.283318    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerStarted","Data":"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"}
Oct 30 12:13:48.706111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:48.706072    4453 patch_prober.go:29] interesting pod/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:13:48.706419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:48.706387    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 12:13:48.706799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:48.706249    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:13:49.708355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:49.705685    4453 patch_prober.go:29] interesting pod/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:13:49.708355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:49.705754    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 12:13:50.714423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:50.702690    4453 patch_prober.go:29] interesting pod/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:13:50.714423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:50.702756    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 12:13:51.707356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:51.702716    4453 patch_prober.go:29] interesting pod/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:13:51.707356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:51.702774    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 12:13:52.672637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:52.672602    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:13:52.709354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:52.706655    4453 patch_prober.go:29] interesting pod/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 12:13:52.709354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:52.706712    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 12:13:53.705421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:53.705392    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:13:53.706435 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:13:53.706410    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:14:07.575571 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:14:07.575532    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:14:12.248852 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.248818    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:14:12.253362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.253324    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.266438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.266414    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"openshift-service-ca.crt\""
Oct 30 12:14:12.275231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.275206    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"kube-root-ca.crt\""
Oct 30 12:14:12.275398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.275224    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846099\""
Oct 30 12:14:12.275398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.275237    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"routing-overlay\""
Oct 30 12:14:12.279784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.279750    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:14:12.344026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.343987    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"payload-processing-external-model-e2e--e097d07a-dockercfg-dl9k7\""
Oct 30 12:14:12.344188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.344086    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846099\"/\"agc-cred-5a665829859a\""
Oct 30 12:14:12.347618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.347587    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.347790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.347627    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.347790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.347663    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8s2\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.347893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.347877    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.454161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.454114    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8s2\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.455762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.455722    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.455928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.455878    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.455928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.455915    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.458306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.458278    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.458488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.458373    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.459026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.459003    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.479190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.479162    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8s2\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\") pod \"payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") " pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.568429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.568324    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:12.872275 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:12.872242    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:14:12.873667 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:14:12.873625    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d60c83c_e724_4002_8597_ad6124a6de0f.slice/crio-26e4bd2a733c8090a010f28d16d7361eddfefda5abd7d8793d48ffc388f007cf WatchSource:0}: Error finding container 26e4bd2a733c8090a010f28d16d7361eddfefda5abd7d8793d48ffc388f007cf: Status 404 returned error can't find the container with id 26e4bd2a733c8090a010f28d16d7361eddfefda5abd7d8793d48ffc388f007cf
Oct 30 12:14:13.470420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:13.470379    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" event={"ID":"8d60c83c-e724-4002-8597-ad6124a6de0f","Type":"ContainerStarted","Data":"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"}
Oct 30 12:14:13.470965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:13.470449    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" event={"ID":"8d60c83c-e724-4002-8597-ad6124a6de0f","Type":"ContainerStarted","Data":"26e4bd2a733c8090a010f28d16d7361eddfefda5abd7d8793d48ffc388f007cf"}
Oct 30 12:14:13.470965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:13.470805    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:13.507532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:13.507482    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" podStartSLOduration=1.507470071 podStartE2EDuration="1.507470071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:13.506235625 +0000 UTC m=+2652.726520534" watchObservedRunningTime="2026-10-30 12:14:13.507470071 +0000 UTC m=+2652.727754984"
Oct 30 12:14:14.722882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.722844    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:14:14.728856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.728825    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.746402 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.746353    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-59b8a9-dockercfg-htngb\""
Oct 30 12:14:14.760514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.760481    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:14:14.786115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.786077    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttb2\" (UniqueName: \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.786284 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.786126    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.807365 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.806640    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-59b8a9\""
Oct 30 12:14:14.888133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.888093    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttb2\" (UniqueName: \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.888133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.888144    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.889156 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.889130    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:14.910340 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:14.910307    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttb2\" (UniqueName: \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\") pod \"payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:15.062624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.062550    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:15.367900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.367872    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:14:15.368990 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:14:15.368963    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1797d832_8456_42cf_83fb_d7fa3493498d.slice/crio-4901348a37dfb09ce0964509973f20028a1d2e434d813592ede894039dd201ca WatchSource:0}: Error finding container 4901348a37dfb09ce0964509973f20028a1d2e434d813592ede894039dd201ca: Status 404 returned error can't find the container with id 4901348a37dfb09ce0964509973f20028a1d2e434d813592ede894039dd201ca
Oct 30 12:14:15.485560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.485519    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" event={"ID":"1797d832-8456-42cf-83fb-d7fa3493498d","Type":"ContainerStarted","Data":"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"}
Oct 30 12:14:15.485716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.485578    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" event={"ID":"1797d832-8456-42cf-83fb-d7fa3493498d","Type":"ContainerStarted","Data":"4901348a37dfb09ce0964509973f20028a1d2e434d813592ede894039dd201ca"}
Oct 30 12:14:15.485716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.485657    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:15.520454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:15.520411    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" podStartSLOduration=1.520398197 podStartE2EDuration="1.520398197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:14:15.520282426 +0000 UTC m=+2654.740567338" watchObservedRunningTime="2026-10-30 12:14:15.520398197 +0000 UTC m=+2654.740683106"
Oct 30 12:14:24.481000 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:24.480972    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:14:26.495953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:26.495926    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:14:37.619849 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:14:37.619805    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:14:38.770695 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:38.770657    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:14:43.771053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:43.771010    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" podUID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" containerName="payload-pre-processing" containerID="cri-o://f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263" gracePeriod=25
Oct 30 12:14:44.053876 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.053843    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:14:44.186440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.186404    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\" (UniqueName: \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\") pod \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") "
Oct 30 12:14:44.186665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.186462    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\") pod \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\" (UID: \"86ac29bc-b19f-4ba4-a56e-c5e7776d0860\") "
Oct 30 12:14:44.186921 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.186890    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume" pod "86ac29bc-b19f-4ba4-a56e-c5e7776d0860" (UID: "86ac29bc-b19f-4ba4-a56e-c5e7776d0860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:14:44.188818 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.188795    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs" pod "86ac29bc-b19f-4ba4-a56e-c5e7776d0860" (UID: "86ac29bc-b19f-4ba4-a56e-c5e7776d0860"). InnerVolumeSpecName "kube-api-access-52hxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:14:44.287857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.287772    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52hxs\" (UniqueName: \"kubernetes.io/projected/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-kube-api-access-52hxs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:14:44.287857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.287806    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ac29bc-b19f-4ba4-a56e-c5e7776d0860-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:14:44.680795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.680758    4453 generic.go:356] "Generic (PLEG): container finished" podID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" containerID="f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263" exitCode=0
Oct 30 12:14:44.680970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.680807    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"
Oct 30 12:14:44.680970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.680835    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" event={"ID":"86ac29bc-b19f-4ba4-a56e-c5e7776d0860","Type":"ContainerDied","Data":"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"}
Oct 30 12:14:44.680970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.680891    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77" event={"ID":"86ac29bc-b19f-4ba4-a56e-c5e7776d0860","Type":"ContainerDied","Data":"35fdb76bf436f86edb4afe1b3342b460267f215467217890a275058b4285d23b"}
Oct 30 12:14:44.680970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.680914    4453 scope.go:122] "RemoveContainer" containerID="f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"
Oct 30 12:14:44.692766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.692743    4453 scope.go:122] "RemoveContainer" containerID="f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"
Oct 30 12:14:44.693896 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:14:44.693852    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263\": container with ID starting with f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263 not found: ID does not exist" containerID="f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"
Oct 30 12:14:44.694035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.693903    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263"} err="failed to get container status \"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263\": rpc error: code = NotFound desc = could not find container \"f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263\": container with ID starting with f07d0b98bce190eba7f611f4342c3d8797ae13720dda4fe80372382372947263 not found: ID does not exist"
Oct 30 12:14:44.733012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.732967    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:14:44.741775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.741745    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-3e23ec-7b4bbd9595-rfh77"]
Oct 30 12:14:44.912989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:14:44.912955    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" path="/var/lib/kubelet/pods/86ac29bc-b19f-4ba4-a56e-c5e7776d0860/volumes"
Oct 30 12:15:00.181813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.181777    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:00.183405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.183370    4453 cpu_manager.go:414] "containerMap: removing container" podUID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" containerName="payload-pre-processing"
Oct 30 12:15:00.183405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.183400    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" containerName="payload-pre-processing"
Oct 30 12:15:00.183642 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.183622    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ac29bc-b19f-4ba4-a56e-c5e7776d0860" containerName="payload-pre-processing"
Oct 30 12:15:00.189506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.189481    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.229399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.229364    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:00.233830 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.231074    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxhx\" (UniqueName: \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.234209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.234170    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.254008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.252139    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:15:00.284623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.284591    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.336267 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.336235    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:00.346496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.344820    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.346496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.346412    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.346496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.345589    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.346782 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.346588    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf65\" (UniqueName: \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.346782 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.346702    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxhx\" (UniqueName: \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.354060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.352476    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:00.354060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.352523    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:00.356196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.355482    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.357272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.357250    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:00.357847 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.357493    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.362037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.362014    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:00.362146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.362125    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.368479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.367752    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:00.368479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.367813    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:00.368479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.367838    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:00.368479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.367878    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:00.368479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.367975    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.369630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.368752    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxhx\" (UniqueName: \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.383465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.383434    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:00.448128 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448039    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhdz\" (UniqueName: \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.448278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448145    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf65\" (UniqueName: \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.448278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448175    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj2q\" (UniqueName: \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.448278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448228    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ptzv\" (UniqueName: \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.448278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448254    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.448460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448282    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.448460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448389    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86nk\" (UniqueName: \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.448460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448430    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.448460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448455    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.448634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.448492    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.449483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.449429    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.469192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.469162    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf65\" (UniqueName: \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\") pod \"maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.549858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.549800    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86nk\" (UniqueName: \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.550082 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.549897    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.550082 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.549928    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.550082 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550034    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhdz\" (UniqueName: \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.550252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550108    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj2q\" (UniqueName: \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.550252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550177    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ptzv\" (UniqueName: \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.550252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550217    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.550493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550252    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.550776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550751    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.550879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550820    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.550961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.550942    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.551031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.551008    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.557616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.557595    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:00.573675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.573646    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86nk\" (UniqueName: \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.575202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.575182    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhdz\" (UniqueName: \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\") pod \"maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.575291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.575222    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj2q\" (UniqueName: \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\") pod \"maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.575291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.575231    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ptzv\" (UniqueName: \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\") pod \"maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.630210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.628241    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:00.674040 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.673998    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:00.708190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.708161    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:00.719746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.719712    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:00.729130 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.729103    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:00.864013 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.863979    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:00.960096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.960063    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:00.960935 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:00.960908    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd265ee56_f9d1_4551_8e31_5dddc2cdb362.slice/crio-6bc38b3ca72606a22ae776b27475855720111dcfe30a7046d4e16e2c1c254906 WatchSource:0}: Error finding container 6bc38b3ca72606a22ae776b27475855720111dcfe30a7046d4e16e2c1c254906: Status 404 returned error can't find the container with id 6bc38b3ca72606a22ae776b27475855720111dcfe30a7046d4e16e2c1c254906
Oct 30 12:15:00.993154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:00.993125    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:00.994028 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:00.994002    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7500c47b_c7fc_42e9_8ba8_daf04b6b0157.slice/crio-5ff0454f88f634c0e69a6559211043bef57d496e991a7c1ca81a9da341942816 WatchSource:0}: Error finding container 5ff0454f88f634c0e69a6559211043bef57d496e991a7c1ca81a9da341942816: Status 404 returned error can't find the container with id 5ff0454f88f634c0e69a6559211043bef57d496e991a7c1ca81a9da341942816
Oct 30 12:15:01.045075 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.045048    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:01.045691 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:01.045665    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d569e1_92ac_48ee_a6c8_ba73e66939ad.slice/crio-a0a967721b7dcb73b686acc3ef61d299267e29340229aef96093fbd21229609e WatchSource:0}: Error finding container a0a967721b7dcb73b686acc3ef61d299267e29340229aef96093fbd21229609e: Status 404 returned error can't find the container with id a0a967721b7dcb73b686acc3ef61d299267e29340229aef96093fbd21229609e
Oct 30 12:15:01.074677 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:01.074649    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c40408_a994_4896_9b32_4239a64be60a.slice/crio-f4b651da97fa4d43877b03309a37be10f2aa91ff54de5e2aa988387a4d2925b0 WatchSource:0}: Error finding container f4b651da97fa4d43877b03309a37be10f2aa91ff54de5e2aa988387a4d2925b0: Status 404 returned error can't find the container with id f4b651da97fa4d43877b03309a37be10f2aa91ff54de5e2aa988387a4d2925b0
Oct 30 12:15:01.074677 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.074662    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:01.080111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.080090    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:01.081953 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:01.081927    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136c848e_1129_456e_9dd3_b208c9951522.slice/crio-8ab3b0db8ff0de9185df33817d255a48df0d89766f7ff142d58fd10e8b6f9ee7 WatchSource:0}: Error finding container 8ab3b0db8ff0de9185df33817d255a48df0d89766f7ff142d58fd10e8b6f9ee7: Status 404 returned error can't find the container with id 8ab3b0db8ff0de9185df33817d255a48df0d89766f7ff142d58fd10e8b6f9ee7
Oct 30 12:15:01.804830 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.804794    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerStarted","Data":"8ab3b0db8ff0de9185df33817d255a48df0d89766f7ff142d58fd10e8b6f9ee7"}
Oct 30 12:15:01.810048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.810012    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerStarted","Data":"bf3703da1821d457d5b19c5d0b0335961f80be28f275c75c91fe152599745ddb"}
Oct 30 12:15:01.812744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.812714    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerStarted","Data":"f4b651da97fa4d43877b03309a37be10f2aa91ff54de5e2aa988387a4d2925b0"}
Oct 30 12:15:01.814448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.814420    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerStarted","Data":"6bc38b3ca72606a22ae776b27475855720111dcfe30a7046d4e16e2c1c254906"}
Oct 30 12:15:01.816958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.816921    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerStarted","Data":"a0a967721b7dcb73b686acc3ef61d299267e29340229aef96093fbd21229609e"}
Oct 30 12:15:01.819739 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:01.819715    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" event={"ID":"7500c47b-c7fc-42e9-8ba8-daf04b6b0157","Type":"ContainerStarted","Data":"5ff0454f88f634c0e69a6559211043bef57d496e991a7c1ca81a9da341942816"}
Oct 30 12:15:05.872497 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:05.872457    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerStarted","Data":"c420c2ae7f4fa8871430a4df71ce9bce5dda970a89969e0b6237d0d2af43aadb"}
Oct 30 12:15:05.874091 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:05.874068    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerStarted","Data":"2e7d1e2fe9fed8e6cf7b5399ea460414d22367e5a4375b3fd26f0c59a984f149"}
Oct 30 12:15:05.909518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:05.909471    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" podStartSLOduration=1.457812604 podStartE2EDuration="5.909456814s" totalImagesPullingTime="4.45164421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:05.906287541 +0000 UTC m=+2705.126572451" watchObservedRunningTime="2026-10-30 12:15:05.909456814 +0000 UTC m=+2705.129741723"
Oct 30 12:15:05.941781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:05.941724    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" podStartSLOduration=1.8938623749999999 podStartE2EDuration="5.941712132s" totalImagesPullingTime="4.047849757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:05.94086234 +0000 UTC m=+2705.161147249" watchObservedRunningTime="2026-10-30 12:15:05.941712132 +0000 UTC m=+2705.161997039"
Oct 30 12:15:06.881798 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.881765    4453 generic.go:356] "Generic (PLEG): container finished" podID="136c848e-1129-456e-9dd3-b208c9951522" containerID="2e7d1e2fe9fed8e6cf7b5399ea460414d22367e5a4375b3fd26f0c59a984f149" exitCode=60
Oct 30 12:15:06.882266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.881854    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerDied","Data":"2e7d1e2fe9fed8e6cf7b5399ea460414d22367e5a4375b3fd26f0c59a984f149"}
Oct 30 12:15:06.882266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.882185    4453 scope.go:122] "RemoveContainer" containerID="2e7d1e2fe9fed8e6cf7b5399ea460414d22367e5a4375b3fd26f0c59a984f149"
Oct 30 12:15:06.883563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.883537    4453 generic.go:356] "Generic (PLEG): container finished" podID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerID="93e2d8f48aee333c6071e3278825373cb38e87e25fa38f24136b47311ff7172f" exitCode=60
Oct 30 12:15:06.883651 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.883623    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerDied","Data":"93e2d8f48aee333c6071e3278825373cb38e87e25fa38f24136b47311ff7172f"}
Oct 30 12:15:06.883863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.883847    4453 scope.go:122] "RemoveContainer" containerID="93e2d8f48aee333c6071e3278825373cb38e87e25fa38f24136b47311ff7172f"
Oct 30 12:15:06.885500 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.885475    4453 generic.go:356] "Generic (PLEG): container finished" podID="72c40408-a994-4896-9b32-4239a64be60a" containerID="3ee71103cef5036b0f30b0038b20e6a8733b635c52ef2e23f7811e728a753120" exitCode=60
Oct 30 12:15:06.885593 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.885548    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerDied","Data":"3ee71103cef5036b0f30b0038b20e6a8733b635c52ef2e23f7811e728a753120"}
Oct 30 12:15:06.885824 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.885808    4453 scope.go:122] "RemoveContainer" containerID="3ee71103cef5036b0f30b0038b20e6a8733b635c52ef2e23f7811e728a753120"
Oct 30 12:15:06.887298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.887274    4453 generic.go:356] "Generic (PLEG): container finished" podID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerID="c420c2ae7f4fa8871430a4df71ce9bce5dda970a89969e0b6237d0d2af43aadb" exitCode=60
Oct 30 12:15:06.887417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.887375    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerDied","Data":"c420c2ae7f4fa8871430a4df71ce9bce5dda970a89969e0b6237d0d2af43aadb"}
Oct 30 12:15:06.887585 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.887566    4453 scope.go:122] "RemoveContainer" containerID="c420c2ae7f4fa8871430a4df71ce9bce5dda970a89969e0b6237d0d2af43aadb"
Oct 30 12:15:06.895808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.895781    4453 generic.go:356] "Generic (PLEG): container finished" podID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerID="fc7b505957be96427e44b776205a6657aea3db810f19d732854024985843bdc9" exitCode=60
Oct 30 12:15:06.896035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.896017    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerDied","Data":"fc7b505957be96427e44b776205a6657aea3db810f19d732854024985843bdc9"}
Oct 30 12:15:06.899801 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.899772    4453 scope.go:122] "RemoveContainer" containerID="fc7b505957be96427e44b776205a6657aea3db810f19d732854024985843bdc9"
Oct 30 12:15:06.903303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.903215    4453 generic.go:356] "Generic (PLEG): container finished" podID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerID="6a8250a0ef79a5a981509b779c8129c63ecc8123f472dddb9d0d321bbb9bb1c7" exitCode=60
Oct 30 12:15:06.903414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.903346    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" event={"ID":"7500c47b-c7fc-42e9-8ba8-daf04b6b0157","Type":"ContainerDied","Data":"6a8250a0ef79a5a981509b779c8129c63ecc8123f472dddb9d0d321bbb9bb1c7"}
Oct 30 12:15:06.903825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:06.903802    4453 scope.go:122] "RemoveContainer" containerID="6a8250a0ef79a5a981509b779c8129c63ecc8123f472dddb9d0d321bbb9bb1c7"
Oct 30 12:15:07.127057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.127031    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:15:07.127349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.127311    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy" containerID="cri-o://1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39" gracePeriod=30
Oct 30 12:15:07.570290 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:07.570247    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:15:07.912625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.912597    4453 generic.go:356] "Generic (PLEG): container finished" podID="72c40408-a994-4896-9b32-4239a64be60a" containerID="f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512" exitCode=60
Oct 30 12:15:07.913029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.912667    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerDied","Data":"f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512"}
Oct 30 12:15:07.913029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.912718    4453 scope.go:122] "RemoveContainer" containerID="3ee71103cef5036b0f30b0038b20e6a8733b635c52ef2e23f7811e728a753120"
Oct 30 12:15:07.913029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.912927    4453 scope.go:122] "RemoveContainer" containerID="f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512"
Oct 30 12:15:07.913240 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:07.913215    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl_odh-ai-gateway-infra(72c40408-a994-4896-9b32-4239a64be60a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" podUID="72c40408-a994-4896-9b32-4239a64be60a"
Oct 30 12:15:07.914880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.914855    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerStarted","Data":"e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"}
Oct 30 12:15:07.916681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.916657    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerStarted","Data":"83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"}
Oct 30 12:15:07.918627 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.918600    4453 generic.go:356] "Generic (PLEG): container finished" podID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerID="131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460" exitCode=60
Oct 30 12:15:07.918721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.918654    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" event={"ID":"7500c47b-c7fc-42e9-8ba8-daf04b6b0157","Type":"ContainerDied","Data":"131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460"}
Oct 30 12:15:07.918918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.918901    4453 scope.go:122] "RemoveContainer" containerID="131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460"
Oct 30 12:15:07.919445 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:07.919156    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg_odh-ai-gateway-infra(7500c47b-c7fc-42e9-8ba8-daf04b6b0157)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157"
Oct 30 12:15:07.921778 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.921760    4453 generic.go:356] "Generic (PLEG): container finished" podID="136c848e-1129-456e-9dd3-b208c9951522" containerID="5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b" exitCode=60
Oct 30 12:15:07.921856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.921842    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerDied","Data":"5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b"}
Oct 30 12:15:07.922077 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.922061    4453 scope.go:122] "RemoveContainer" containerID="5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b"
Oct 30 12:15:07.922285 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:07.922259    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr_odh-ai-gateway-infra(136c848e-1129-456e-9dd3-b208c9951522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" podUID="136c848e-1129-456e-9dd3-b208c9951522"
Oct 30 12:15:07.923890 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.923865    4453 generic.go:356] "Generic (PLEG): container finished" podID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerID="857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc" exitCode=60
Oct 30 12:15:07.924006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.923921    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerDied","Data":"857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc"}
Oct 30 12:15:07.924207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:07.924189    4453 scope.go:122] "RemoveContainer" containerID="857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc"
Oct 30 12:15:07.924469 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:07.924447    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn_odh-ai-gateway-infra(b073803c-ec84-4dcf-a7a0-6155693811c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5"
Oct 30 12:15:08.091565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.091503    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" podStartSLOduration=3.723636414 podStartE2EDuration="8.091484602s" totalImagesPullingTime="4.367848188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:08.088643095 +0000 UTC m=+2707.308928009" watchObservedRunningTime="2026-10-30 12:15:08.091484602 +0000 UTC m=+2707.311769517"
Oct 30 12:15:08.193938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.193916    4453 scope.go:122] "RemoveContainer" containerID="6a8250a0ef79a5a981509b779c8129c63ecc8123f472dddb9d0d321bbb9bb1c7"
Oct 30 12:15:08.207026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.207000    4453 scope.go:122] "RemoveContainer" containerID="2e7d1e2fe9fed8e6cf7b5399ea460414d22367e5a4375b3fd26f0c59a984f149"
Oct 30 12:15:08.221321 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.221291    4453 scope.go:122] "RemoveContainer" containerID="93e2d8f48aee333c6071e3278825373cb38e87e25fa38f24136b47311ff7172f"
Oct 30 12:15:08.931940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.931905    4453 scope.go:122] "RemoveContainer" containerID="5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b"
Oct 30 12:15:08.932391 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.932209    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr_odh-ai-gateway-infra(136c848e-1129-456e-9dd3-b208c9951522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" podUID="136c848e-1129-456e-9dd3-b208c9951522"
Oct 30 12:15:08.934259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.934238    4453 scope.go:122] "RemoveContainer" containerID="857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc"
Oct 30 12:15:08.934570 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.934544    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn_odh-ai-gateway-infra(b073803c-ec84-4dcf-a7a0-6155693811c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5"
Oct 30 12:15:08.936448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.936130    4453 scope.go:122] "RemoveContainer" containerID="f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512"
Oct 30 12:15:08.936448 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.936404    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl_odh-ai-gateway-infra(72c40408-a994-4896-9b32-4239a64be60a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" podUID="72c40408-a994-4896-9b32-4239a64be60a"
Oct 30 12:15:08.941868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.941841    4453 generic.go:356] "Generic (PLEG): container finished" podID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerID="e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb" exitCode=60
Oct 30 12:15:08.942057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.941923    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerDied","Data":"e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"}
Oct 30 12:15:08.942057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.941981    4453 scope.go:122] "RemoveContainer" containerID="c420c2ae7f4fa8871430a4df71ce9bce5dda970a89969e0b6237d0d2af43aadb"
Oct 30 12:15:08.942176 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.942148    4453 scope.go:122] "RemoveContainer" containerID="e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"
Oct 30 12:15:08.942421 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.942389    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684_odh-ai-gateway-infra(d265ee56-f9d1-4551-8e31-5dddc2cdb362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362"
Oct 30 12:15:08.943821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.943799    4453 generic.go:356] "Generic (PLEG): container finished" podID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerID="83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f" exitCode=60
Oct 30 12:15:08.943940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.943864    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerDied","Data":"83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"}
Oct 30 12:15:08.944109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.944092    4453 scope.go:122] "RemoveContainer" containerID="83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"
Oct 30 12:15:08.944364 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.944314    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q_odh-ai-gateway-infra(d3d569e1-92ac-48ee-a6c8-ba73e66939ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad"
Oct 30 12:15:08.945799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.945780    4453 scope.go:122] "RemoveContainer" containerID="131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460"
Oct 30 12:15:08.946002 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:08.945981    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg_odh-ai-gateway-infra(7500c47b-c7fc-42e9-8ba8-daf04b6b0157)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157"
Oct 30 12:15:08.956736 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:08.956716    4453 scope.go:122] "RemoveContainer" containerID="fc7b505957be96427e44b776205a6657aea3db810f19d732854024985843bdc9"
Oct 30 12:15:09.551822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:09.551780    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:15:09.552130 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:09.552071    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy" containerID="cri-o://5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626" gracePeriod=30
Oct 30 12:15:09.953587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:09.953559    4453 scope.go:122] "RemoveContainer" containerID="e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"
Oct 30 12:15:09.954059 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:09.953920    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684_odh-ai-gateway-infra(d265ee56-f9d1-4551-8e31-5dddc2cdb362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362"
Oct 30 12:15:09.955287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:09.955267    4453 scope.go:122] "RemoveContainer" containerID="83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"
Oct 30 12:15:09.955578 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:09.955559    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q_odh-ai-gateway-infra(d3d569e1-92ac-48ee-a6c8-ba73e66939ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad"
Oct 30 12:15:12.401571 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.401504    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:15:12.501364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501291    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501362    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501443    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501566    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501611    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501659    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501699    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501767    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.501954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501775    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:12.501954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501803    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:12.501954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.501911    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\") pod \"79716152-156c-4d1e-9fc2-465dc9e6995c\" (UID: \"79716152-156c-4d1e-9fc2-465dc9e6995c\") "
Oct 30 12:15:12.502114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502067    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:12.502114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502083    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:12.502263 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502232    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:12.502532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502510    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.502613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502538    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.502613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502551    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.502613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502565    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79716152-156c-4d1e-9fc2-465dc9e6995c-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.502613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.502579    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.504255 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.504219    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:12.504386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.504288    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "kube-api-access-q5pjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:12.504599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.504576    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:12.504686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.504671    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo" pod "79716152-156c-4d1e-9fc2-465dc9e6995c" (UID: "79716152-156c-4d1e-9fc2-465dc9e6995c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:15:12.603582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.603549    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.603582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.603575    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pjk\" (UniqueName: \"kubernetes.io/projected/79716152-156c-4d1e-9fc2-465dc9e6995c-kube-api-access-q5pjk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.603582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.603587    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.603822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.603596    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79716152-156c-4d1e-9fc2-465dc9e6995c-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:12.974387 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.974354    4453 generic.go:356] "Generic (PLEG): container finished" podID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerID="1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39" exitCode=0
Oct 30 12:15:12.974541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.974406    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"
Oct 30 12:15:12.974541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.974442    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" event={"ID":"79716152-156c-4d1e-9fc2-465dc9e6995c","Type":"ContainerDied","Data":"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"}
Oct 30 12:15:12.974541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.974494    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f" event={"ID":"79716152-156c-4d1e-9fc2-465dc9e6995c","Type":"ContainerDied","Data":"ab7bbb8f19e58a523a621c0cab489f73c05a106b9ce49ff4ae6f8ce22d49d3ce"}
Oct 30 12:15:12.974541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.974516    4453 scope.go:122] "RemoveContainer" containerID="1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"
Oct 30 12:15:12.984402 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.984382    4453 scope.go:122] "RemoveContainer" containerID="1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"
Oct 30 12:15:12.984679 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:12.984659    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39\": container with ID starting with 1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39 not found: ID does not exist" containerID="1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"
Oct 30 12:15:12.984729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:12.984685    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39"} err="failed to get container status \"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39\": rpc error: code = NotFound desc = could not find container \"1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39\": container with ID starting with 1bb42b771001b3e9d18e8c27dd8d70f4470491d12dcb21c4829c374b493c3f39 not found: ID does not exist"
Oct 30 12:15:13.021430 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:13.021403    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:15:13.031139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:13.031113    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-spb7f"]
Oct 30 12:15:14.833610 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.833550    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:15:14.916664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.916627    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" path="/var/lib/kubelet/pods/79716152-156c-4d1e-9fc2-465dc9e6995c/volumes"
Oct 30 12:15:14.930724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.930691    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.930900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.930747    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.930900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.930823    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.930900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.930866    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.931061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.930913    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.931118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931059    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.931171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931118    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.931222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931164    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.931222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931199    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\") pod \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\" (UID: \"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c\") "
Oct 30 12:15:14.932380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931763    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:14.932380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931897    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:14.932380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.931911    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:14.932380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.932345    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:14.933004 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.932982    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:14.935531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.935503    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:15:14.935635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.935546    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:14.936585 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.936558    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "kube-api-access-4vhxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:14.937392 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.937371    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo" pod "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" (UID: "7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:15:14.989544 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.989507    4453 generic.go:356] "Generic (PLEG): container finished" podID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerID="5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626" exitCode=0
Oct 30 12:15:14.989730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.989559    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"
Oct 30 12:15:14.989730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.989588    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerDied","Data":"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"}
Oct 30 12:15:14.989730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.989649    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28" event={"ID":"7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c","Type":"ContainerDied","Data":"d73bfdc705cfa45f0aec9519f35a06d895f7e10c7c4433fa383b0fcab1a9c8a6"}
Oct 30 12:15:14.989730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:14.989677    4453 scope.go:122] "RemoveContainer" containerID="5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"
Oct 30 12:15:15.000378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.000362    4453 scope.go:122] "RemoveContainer" containerID="5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"
Oct 30 12:15:15.009744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.009725    4453 scope.go:122] "RemoveContainer" containerID="5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"
Oct 30 12:15:15.010027 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:15.010007    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626\": container with ID starting with 5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626 not found: ID does not exist" containerID="5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"
Oct 30 12:15:15.010092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.010037    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626"} err="failed to get container status \"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626\": rpc error: code = NotFound desc = could not find container \"5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626\": container with ID starting with 5fb8a707600ba09f4e578ef3603697e69ff4d7822ef4b8c2e6a494e1b4222626 not found: ID does not exist"
Oct 30 12:15:15.010092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.010056    4453 scope.go:122] "RemoveContainer" containerID="5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"
Oct 30 12:15:15.010305 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:15.010286    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe\": container with ID starting with 5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe not found: ID does not exist" containerID="5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"
Oct 30 12:15:15.010434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.010309    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe"} err="failed to get container status \"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe\": rpc error: code = NotFound desc = could not find container \"5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe\": container with ID starting with 5fdd1b60a1ce3d5bbc8b07222aea047a16957d5cb1d3bce69be36f0185980bbe not found: ID does not exist"
Oct 30 12:15:15.032959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032931    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.032959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032955    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.032959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032965    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.032959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032973    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.033273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032982    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.033273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032990    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vhxw\" (UniqueName: \"kubernetes.io/projected/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-kube-api-access-4vhxw\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.033273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.032999    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.033273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.033007    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.033273 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.033015    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:15.036147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.036118    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:15:15.044931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:15.044906    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3e23ec-openshift-default-5dcb78b869-5ww28"]
Oct 30 12:15:16.912031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:16.911995    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" path="/var/lib/kubelet/pods/7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c/volumes"
Oct 30 12:15:18.905045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:18.905002    4453 scope.go:122] "RemoveContainer" containerID="131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460"
Oct 30 12:15:18.905533 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:18.905224    4453 scope.go:122] "RemoveContainer" containerID="e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"
Oct 30 12:15:19.904655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:19.904622    4453 scope.go:122] "RemoveContainer" containerID="857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc"
Oct 30 12:15:19.904962 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:19.904941    4453 scope.go:122] "RemoveContainer" containerID="83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"
Oct 30 12:15:19.905180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:19.905158    4453 scope.go:122] "RemoveContainer" containerID="f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512"
Oct 30 12:15:20.028087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.028058    4453 generic.go:356] "Generic (PLEG): container finished" podID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerID="3ed2ae7e0eebf9d2effeb2962481a6c9d9b3834cfba89748fc4c67101a5cf0fb" exitCode=60
Oct 30 12:15:20.028241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.028131    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerDied","Data":"3ed2ae7e0eebf9d2effeb2962481a6c9d9b3834cfba89748fc4c67101a5cf0fb"}
Oct 30 12:15:20.028241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.028195    4453 scope.go:122] "RemoveContainer" containerID="e7436cf049d0ff2866f1c71e8d4ade886cc57f7955c8914d5e5e7a0d811ddecb"
Oct 30 12:15:20.028611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.028588    4453 scope.go:122] "RemoveContainer" containerID="3ed2ae7e0eebf9d2effeb2962481a6c9d9b3834cfba89748fc4c67101a5cf0fb"
Oct 30 12:15:20.029108 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:20.029083    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684_odh-ai-gateway-infra(d265ee56-f9d1-4551-8e31-5dddc2cdb362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362"
Oct 30 12:15:20.031385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.031362    4453 generic.go:356] "Generic (PLEG): container finished" podID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerID="f5933e964aef075ae3767cbdf990546432f5e003649f3f1da344aaf4cb4f754f" exitCode=60
Oct 30 12:15:20.031496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.031428    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" event={"ID":"7500c47b-c7fc-42e9-8ba8-daf04b6b0157","Type":"ContainerDied","Data":"f5933e964aef075ae3767cbdf990546432f5e003649f3f1da344aaf4cb4f754f"}
Oct 30 12:15:20.031822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.031801    4453 scope.go:122] "RemoveContainer" containerID="f5933e964aef075ae3767cbdf990546432f5e003649f3f1da344aaf4cb4f754f"
Oct 30 12:15:20.032162 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:20.032140    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg_odh-ai-gateway-infra(7500c47b-c7fc-42e9-8ba8-daf04b6b0157)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157"
Oct 30 12:15:20.122872 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.121877    4453 scope.go:122] "RemoveContainer" containerID="131a128bf17a8d7846ed5aba25a48ebb64e03258227ec6462fe7d21d9e9d1460"
Oct 30 12:15:20.911713 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:20.911684    4453 scope.go:122] "RemoveContainer" containerID="5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b"
Oct 30 12:15:21.033576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.033543    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:15:21.034224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034205    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034222    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034238    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034244    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034254    4453 cpu_manager.go:414] "containerMap: removing container" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034259    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034370    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034381    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d61d5ef-9e3f-49b3-8049-a6bb8fe4e69c" containerName="istio-proxy"
Oct 30 12:15:21.034487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.034389    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="79716152-156c-4d1e-9fc2-465dc9e6995c" containerName="istio-proxy"
Oct 30 12:15:21.038060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.038038    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.041384 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.041359    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerStarted","Data":"4bda56faf02664a6f94db605f3c5449e2c8b7fe5d0e0405346b477638e154d12"}
Oct 30 12:15:21.043499 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.043476    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerStarted","Data":"c4fe6de1b7ba8098554f2f1b3f345caac58ea1c59203bededbaa2037d8ae9cc9"}
Oct 30 12:15:21.046955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.046935    4453 generic.go:356] "Generic (PLEG): container finished" podID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerID="090dbc6992f2c9c1a2be525583c09b64b7fbf8c4e42c2e7571dc956fbaab47e8" exitCode=60
Oct 30 12:15:21.047068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.046987    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerDied","Data":"090dbc6992f2c9c1a2be525583c09b64b7fbf8c4e42c2e7571dc956fbaab47e8"}
Oct 30 12:15:21.047068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.047041    4453 scope.go:122] "RemoveContainer" containerID="83b7ca04e82773997fe5ec71b50d8076ed939494786b80ab22bea0619435a14f"
Oct 30 12:15:21.047376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.047361    4453 scope.go:122] "RemoveContainer" containerID="090dbc6992f2c9c1a2be525583c09b64b7fbf8c4e42c2e7571dc956fbaab47e8"
Oct 30 12:15:21.047708 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:21.047682    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q_odh-ai-gateway-infra(d3d569e1-92ac-48ee-a6c8-ba73e66939ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad"
Oct 30 12:15:21.052813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.052796    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"e2e-cmp-legacy-e0015e-provider-cred\""
Oct 30 12:15:21.055608 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.055588    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"routing-overlay\""
Oct 30 12:15:21.062618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.062596    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f8f132\""
Oct 30 12:15:21.062737 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.062668    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"openshift-service-ca.crt\""
Oct 30 12:15:21.068780 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.066788    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"kube-root-ca.crt\""
Oct 30 12:15:21.075484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.075460    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:15:21.081842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.081819    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:21.115141 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.115087    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" podStartSLOduration=16.584236577 podStartE2EDuration="21.115073598s" totalImagesPullingTime="4.530837021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:21.114420885 +0000 UTC m=+2720.334705797" watchObservedRunningTime="2026-10-30 12:15:21.115073598 +0000 UTC m=+2720.335358507"
Oct 30 12:15:21.127870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.127834    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:21.202258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.202225    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-8wcx6\""
Oct 30 12:15:21.245424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.215363    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dff\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.245424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.215508    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.245424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.215528    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.245424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.215609    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.262442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.262385    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" podStartSLOduration=16.935587418 podStartE2EDuration="21.262367953s" totalImagesPullingTime="4.326780535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:21.261381294 +0000 UTC m=+2720.481666201" watchObservedRunningTime="2026-10-30 12:15:21.262367953 +0000 UTC m=+2720.482652863"
Oct 30 12:15:21.321051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.320199    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.321051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.320317    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dff\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.321051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.320497    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.321051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.320533    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.322405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.321597    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.322554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.322518    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.328148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.328110    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.343750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.343721    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dff\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.373878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.373848    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:21.379180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.379152    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:21.427665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.427638    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:21.523879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.523846    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\") pod \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") "
Oct 30 12:15:21.524120 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524098    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\" (UniqueName: \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\") pod \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") "
Oct 30 12:15:21.524192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524172    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\") pod \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\" (UID: \"7500c47b-c7fc-42e9-8ba8-daf04b6b0157\") "
Oct 30 12:15:21.524248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524198    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\" (UniqueName: \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\") pod \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\" (UID: \"d265ee56-f9d1-4551-8e31-5dddc2cdb362\") "
Oct 30 12:15:21.524837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524395    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca" pod "d265ee56-f9d1-4551-8e31-5dddc2cdb362" (UID: "d265ee56-f9d1-4551-8e31-5dddc2cdb362"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:21.524837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524654    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca" pod "7500c47b-c7fc-42e9-8ba8-daf04b6b0157" (UID: "7500c47b-c7fc-42e9-8ba8-daf04b6b0157"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:21.524999 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524865    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.524999 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.524891    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d265ee56-f9d1-4551-8e31-5dddc2cdb362-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.526904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.526881    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk" pod "7500c47b-c7fc-42e9-8ba8-daf04b6b0157" (UID: "7500c47b-c7fc-42e9-8ba8-daf04b6b0157"). InnerVolumeSpecName "kube-api-access-v86nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:21.527009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.526916    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65" pod "d265ee56-f9d1-4551-8e31-5dddc2cdb362" (UID: "d265ee56-f9d1-4551-8e31-5dddc2cdb362"). InnerVolumeSpecName "kube-api-access-4xf65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:21.626534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.626501    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86nk\" (UniqueName: \"kubernetes.io/projected/7500c47b-c7fc-42e9-8ba8-daf04b6b0157-kube-api-access-v86nk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.626534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.626530    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xf65\" (UniqueName: \"kubernetes.io/projected/d265ee56-f9d1-4551-8e31-5dddc2cdb362-kube-api-access-4xf65\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:21.700354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:21.700298    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:15:21.700984 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:21.700946    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ce321_2f6f_4528_af22_e14957d333c1.slice/crio-b8d1657461e9683977017d60affce7e3b88d6ec67d5b5d887aa25a6a6d7891b0 WatchSource:0}: Error finding container b8d1657461e9683977017d60affce7e3b88d6ec67d5b5d887aa25a6a6d7891b0: Status 404 returned error can't find the container with id b8d1657461e9683977017d60affce7e3b88d6ec67d5b5d887aa25a6a6d7891b0
Oct 30 12:15:22.082279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.082147    4453 generic.go:356] "Generic (PLEG): container finished" podID="72c40408-a994-4896-9b32-4239a64be60a" containerID="c4fe6de1b7ba8098554f2f1b3f345caac58ea1c59203bededbaa2037d8ae9cc9" exitCode=60
Oct 30 12:15:22.082929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.082293    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerDied","Data":"c4fe6de1b7ba8098554f2f1b3f345caac58ea1c59203bededbaa2037d8ae9cc9"}
Oct 30 12:15:22.083141 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.083108    4453 scope.go:122] "RemoveContainer" containerID="f346ee7c7ff12f20c8069aed816f4c532f2df1370235b60a4b1be0d445506512"
Oct 30 12:15:22.083944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.083921    4453 scope.go:122] "RemoveContainer" containerID="c4fe6de1b7ba8098554f2f1b3f345caac58ea1c59203bededbaa2037d8ae9cc9"
Oct 30 12:15:22.084386 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:22.084360    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl_odh-ai-gateway-infra(72c40408-a994-4896-9b32-4239a64be60a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" podUID="72c40408-a994-4896-9b32-4239a64be60a"
Oct 30 12:15:22.085305 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.085285    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"
Oct 30 12:15:22.085631 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.085595    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684" event={"ID":"d265ee56-f9d1-4551-8e31-5dddc2cdb362","Type":"ContainerDied","Data":"6bc38b3ca72606a22ae776b27475855720111dcfe30a7046d4e16e2c1c254906"}
Oct 30 12:15:22.092136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.092110    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"
Oct 30 12:15:22.092251 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.092104    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg" event={"ID":"7500c47b-c7fc-42e9-8ba8-daf04b6b0157","Type":"ContainerDied","Data":"5ff0454f88f634c0e69a6559211043bef57d496e991a7c1ca81a9da341942816"}
Oct 30 12:15:22.097751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.097485    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" event={"ID":"b96ce321-2f6f-4528-af22-e14957d333c1","Type":"ContainerStarted","Data":"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"}
Oct 30 12:15:22.097751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.097536    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" event={"ID":"b96ce321-2f6f-4528-af22-e14957d333c1","Type":"ContainerStarted","Data":"b8d1657461e9683977017d60affce7e3b88d6ec67d5b5d887aa25a6a6d7891b0"}
Oct 30 12:15:22.097751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.097645    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:22.099654 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.099619    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerStarted","Data":"79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836"}
Oct 30 12:15:22.100009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.099990    4453 scope.go:122] "RemoveContainer" containerID="79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836"
Oct 30 12:15:22.100285 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:22.100262    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr_odh-ai-gateway-infra(136c848e-1129-456e-9dd3-b208c9951522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" podUID="136c848e-1129-456e-9dd3-b208c9951522"
Oct 30 12:15:22.102027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.102006    4453 generic.go:356] "Generic (PLEG): container finished" podID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerID="4bda56faf02664a6f94db605f3c5449e2c8b7fe5d0e0405346b477638e154d12" exitCode=60
Oct 30 12:15:22.102112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.102044    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerDied","Data":"4bda56faf02664a6f94db605f3c5449e2c8b7fe5d0e0405346b477638e154d12"}
Oct 30 12:15:22.102555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.102458    4453 scope.go:122] "RemoveContainer" containerID="4bda56faf02664a6f94db605f3c5449e2c8b7fe5d0e0405346b477638e154d12"
Oct 30 12:15:22.102728 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:22.102670    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn_odh-ai-gateway-infra(b073803c-ec84-4dcf-a7a0-6155693811c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5"
Oct 30 12:15:22.129437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.128812    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:22.155261 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.155234    4453 scope.go:122] "RemoveContainer" containerID="3ed2ae7e0eebf9d2effeb2962481a6c9d9b3834cfba89748fc4c67101a5cf0fb"
Oct 30 12:15:22.165261 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.165240    4453 scope.go:122] "RemoveContainer" containerID="f5933e964aef075ae3767cbdf990546432f5e003649f3f1da344aaf4cb4f754f"
Oct 30 12:15:22.173836 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.173809    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:22.174950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.174927    4453 scope.go:122] "RemoveContainer" containerID="857b4e09f9c3a6353abbbe0d6d16427016ae7084954f06b0b57a29f7ce77eedc"
Oct 30 12:15:22.177409 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.177359    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" podStartSLOduration=1.177346184 podStartE2EDuration="1.177346184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:22.173849775 +0000 UTC m=+2721.394134685" watchObservedRunningTime="2026-10-30 12:15:22.177346184 +0000 UTC m=+2721.397631085"
Oct 30 12:15:22.273193 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.273115    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:22.337669 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.337640    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:22.343391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.343365    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889375-zzzcg"]
Oct 30 12:15:22.362550 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.362526    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:22.371880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.371857    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:22.375282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.375256    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-846099-29889375-cb684"]
Oct 30 12:15:22.456215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.456181    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\" (UniqueName: \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\") pod \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") "
Oct 30 12:15:22.456379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.456228    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\") pod \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\" (UID: \"d3d569e1-92ac-48ee-a6c8-ba73e66939ad\") "
Oct 30 12:15:22.456689 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.456665    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca" pod "d3d569e1-92ac-48ee-a6c8-ba73e66939ad" (UID: "d3d569e1-92ac-48ee-a6c8-ba73e66939ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:22.456888 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.456868    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:22.458541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.458510    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q" pod "d3d569e1-92ac-48ee-a6c8-ba73e66939ad" (UID: "d3d569e1-92ac-48ee-a6c8-ba73e66939ad"). InnerVolumeSpecName "kube-api-access-7xj2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:22.557609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.557523    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xj2q\" (UniqueName: \"kubernetes.io/projected/d3d569e1-92ac-48ee-a6c8-ba73e66939ad-kube-api-access-7xj2q\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:22.913804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.913770    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" path="/var/lib/kubelet/pods/7500c47b-c7fc-42e9-8ba8-daf04b6b0157/volumes"
Oct 30 12:15:22.914253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:22.914239    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" path="/var/lib/kubelet/pods/d265ee56-f9d1-4551-8e31-5dddc2cdb362/volumes"
Oct 30 12:15:23.111133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.111095    4453 generic.go:356] "Generic (PLEG): container finished" podID="136c848e-1129-456e-9dd3-b208c9951522" containerID="79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836" exitCode=60
Oct 30 12:15:23.111622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.111128    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerDied","Data":"79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836"}
Oct 30 12:15:23.111622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.111206    4453 scope.go:122] "RemoveContainer" containerID="5c2aea3004b5e3ba2f654badcff9e01027b25a88cc6be3d0b31cf2893fd8b63b"
Oct 30 12:15:23.111622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.111613    4453 scope.go:122] "RemoveContainer" containerID="79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836"
Oct 30 12:15:23.111939 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:23.111916    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr_odh-ai-gateway-infra(136c848e-1129-456e-9dd3-b208c9951522)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" podUID="136c848e-1129-456e-9dd3-b208c9951522"
Oct 30 12:15:23.125622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.125593    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q" event={"ID":"d3d569e1-92ac-48ee-a6c8-ba73e66939ad","Type":"ContainerDied","Data":"a0a967721b7dcb73b686acc3ef61d299267e29340229aef96093fbd21229609e"}
Oct 30 12:15:23.125717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.125611    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"
Oct 30 12:15:23.136270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.136249    4453 scope.go:122] "RemoveContainer" containerID="090dbc6992f2c9c1a2be525583c09b64b7fbf8c4e42c2e7571dc956fbaab47e8"
Oct 30 12:15:23.226307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.226274    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:23.231636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.231347    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4f7443-29889375-qck4q"]
Oct 30 12:15:23.271316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.271279    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:23.304520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.303419    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:23.327746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.327720    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:23.374637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.374603    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\") pod \"b073803c-ec84-4dcf-a7a0-6155693811c5\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") "
Oct 30 12:15:23.374816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.374667    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\" (UniqueName: \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\") pod \"72c40408-a994-4896-9b32-4239a64be60a\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") "
Oct 30 12:15:23.374816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.374725    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\" (UniqueName: \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\") pod \"b073803c-ec84-4dcf-a7a0-6155693811c5\" (UID: \"b073803c-ec84-4dcf-a7a0-6155693811c5\") "
Oct 30 12:15:23.374953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.374817    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\") pod \"72c40408-a994-4896-9b32-4239a64be60a\" (UID: \"72c40408-a994-4896-9b32-4239a64be60a\") "
Oct 30 12:15:23.375055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.375001    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca" pod "b073803c-ec84-4dcf-a7a0-6155693811c5" (UID: "b073803c-ec84-4dcf-a7a0-6155693811c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.375445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.375409    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073803c-ec84-4dcf-a7a0-6155693811c5-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.375445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.375419    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca" pod "72c40408-a994-4896-9b32-4239a64be60a" (UID: "72c40408-a994-4896-9b32-4239a64be60a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:23.380293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.380267    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz" pod "72c40408-a994-4896-9b32-4239a64be60a" (UID: "72c40408-a994-4896-9b32-4239a64be60a"). InnerVolumeSpecName "kube-api-access-vbhdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.380446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.380324    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx" pod "b073803c-ec84-4dcf-a7a0-6155693811c5" (UID: "b073803c-ec84-4dcf-a7a0-6155693811c5"). InnerVolumeSpecName "kube-api-access-rcxhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:23.476094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.476032    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhdz\" (UniqueName: \"kubernetes.io/projected/72c40408-a994-4896-9b32-4239a64be60a-kube-api-access-vbhdz\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.476094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.476060    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxhx\" (UniqueName: \"kubernetes.io/projected/b073803c-ec84-4dcf-a7a0-6155693811c5-kube-api-access-rcxhx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:23.476094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:23.476070    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c40408-a994-4896-9b32-4239a64be60a-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:24.135970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.135940    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"
Oct 30 12:15:24.135970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.135951    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn" event={"ID":"b073803c-ec84-4dcf-a7a0-6155693811c5","Type":"ContainerDied","Data":"bf3703da1821d457d5b19c5d0b0335961f80be28f275c75c91fe152599745ddb"}
Oct 30 12:15:24.136447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.136018    4453 scope.go:122] "RemoveContainer" containerID="4bda56faf02664a6f94db605f3c5449e2c8b7fe5d0e0405346b477638e154d12"
Oct 30 12:15:24.137305 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.137272    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"
Oct 30 12:15:24.137426 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.137357    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl" event={"ID":"72c40408-a994-4896-9b32-4239a64be60a","Type":"ContainerDied","Data":"f4b651da97fa4d43877b03309a37be10f2aa91ff54de5e2aa988387a4d2925b0"}
Oct 30 12:15:24.146952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.146884    4453 scope.go:122] "RemoveContainer" containerID="c4fe6de1b7ba8098554f2f1b3f345caac58ea1c59203bededbaa2037d8ae9cc9"
Oct 30 12:15:24.205861 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.205821    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:24.212895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.212639    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889375-wgxtn"]
Oct 30 12:15:24.239847 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.239815    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:24.245211 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.245183    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f8f132-29889375-wkfcl"]
Oct 30 12:15:24.292956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.292931    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:24.387364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.387248    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\" (UniqueName: \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\") pod \"136c848e-1129-456e-9dd3-b208c9951522\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") "
Oct 30 12:15:24.387534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.387452    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\") pod \"136c848e-1129-456e-9dd3-b208c9951522\" (UID: \"136c848e-1129-456e-9dd3-b208c9951522\") "
Oct 30 12:15:24.387855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.387827    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca" pod "136c848e-1129-456e-9dd3-b208c9951522" (UID: "136c848e-1129-456e-9dd3-b208c9951522"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:24.389304 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.389285    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv" pod "136c848e-1129-456e-9dd3-b208c9951522" (UID: "136c848e-1129-456e-9dd3-b208c9951522"). InnerVolumeSpecName "kube-api-access-2ptzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:24.488776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.488745    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ptzv\" (UniqueName: \"kubernetes.io/projected/136c848e-1129-456e-9dd3-b208c9951522-kube-api-access-2ptzv\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:24.488776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.488772    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/136c848e-1129-456e-9dd3-b208c9951522-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:24.913765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.913733    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c40408-a994-4896-9b32-4239a64be60a" path="/var/lib/kubelet/pods/72c40408-a994-4896-9b32-4239a64be60a/volumes"
Oct 30 12:15:24.914181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.914166    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" path="/var/lib/kubelet/pods/b073803c-ec84-4dcf-a7a0-6155693811c5/volumes"
Oct 30 12:15:24.914551 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:24.914538    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" path="/var/lib/kubelet/pods/d3d569e1-92ac-48ee-a6c8-ba73e66939ad/volumes"
Oct 30 12:15:25.147503 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:25.147472    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"
Oct 30 12:15:25.147961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:25.147507    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr" event={"ID":"136c848e-1129-456e-9dd3-b208c9951522","Type":"ContainerDied","Data":"8ab3b0db8ff0de9185df33817d255a48df0d89766f7ff142d58fd10e8b6f9ee7"}
Oct 30 12:15:25.147961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:25.147575    4453 scope.go:122] "RemoveContainer" containerID="79950c9e0e882b7f2cd27664387fcb2ebfc567302516b2e990bb7e2f80b72836"
Oct 30 12:15:25.192893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:25.192820    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:25.201845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:25.201822    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2bc4e2-29889375-mx5qr"]
Oct 30 12:15:26.911486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:26.911451    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136c848e-1129-456e-9dd3-b208c9951522" path="/var/lib/kubelet/pods/136c848e-1129-456e-9dd3-b208c9951522/volumes"
Oct 30 12:15:28.223615 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.223583    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:15:28.224171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224158    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224172    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224182    4453 cpu_manager.go:414] "containerMap: removing container" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224187    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224194    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224199    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224208    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.224216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224213    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224221    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224225    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224235    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224240    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224245    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224249    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224259    4453 cpu_manager.go:414] "containerMap: removing container" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224263    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224273    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224277    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224284    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224289    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224297    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224301    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224315    4453 cpu_manager.go:414] "containerMap: removing container" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224319    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224325    4453 cpu_manager.go:414] "containerMap: removing container" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224343    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224364    4453 cpu_manager.go:414] "containerMap: removing container" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224369    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224376    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224380    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224387    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224393    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224480    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.224476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224489    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224498    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224503    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224509    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224514    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224519    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224524    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224531    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224535    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224542    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224548    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.225264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.224553    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="136c848e-1129-456e-9dd3-b208c9951522" containerName="cleanup"
Oct 30 12:15:28.229142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.229127    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.241231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.241210    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-67e819-dockercfg-8ftcz\""
Oct 30 12:15:28.241391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.241284    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-67e819\""
Oct 30 12:15:28.248432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.248409    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:15:28.335436 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.333430    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.335436 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.333617    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-244d7\" (UniqueName: \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.434776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.434740    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-244d7\" (UniqueName: \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.434969 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.434876    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.435611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.435587    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.457993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.457965    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-244d7\" (UniqueName: \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\") pod \"payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.542701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.542632    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:28.843566 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:28.843537    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:15:28.844538 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:28.844511    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9148c9db_ae77_4091_a0ef_31af5a722d08.slice/crio-11573a15c33b72a40b07c0c25c86a022f1e5586cea3dd780f9c3ad542111b46c WatchSource:0}: Error finding container 11573a15c33b72a40b07c0c25c86a022f1e5586cea3dd780f9c3ad542111b46c: Status 404 returned error can't find the container with id 11573a15c33b72a40b07c0c25c86a022f1e5586cea3dd780f9c3ad542111b46c
Oct 30 12:15:29.180554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:29.180504    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" event={"ID":"9148c9db-ae77-4091-a0ef-31af5a722d08","Type":"ContainerStarted","Data":"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"}
Oct 30 12:15:29.180752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:29.180571    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:29.180752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:29.180595    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" event={"ID":"9148c9db-ae77-4091-a0ef-31af5a722d08","Type":"ContainerStarted","Data":"11573a15c33b72a40b07c0c25c86a022f1e5586cea3dd780f9c3ad542111b46c"}
Oct 30 12:15:29.221172 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:29.221123    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" podStartSLOduration=1.221109277 podStartE2EDuration="1.221109277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:29.218549821 +0000 UTC m=+2728.438834731" watchObservedRunningTime="2026-10-30 12:15:29.221109277 +0000 UTC m=+2728.441394210"
Oct 30 12:15:33.130243 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:33.130212    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:15:37.566231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:37.566089    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:15:37.575361 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:37.574349    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:15:40.188935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.188905    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:15:40.272001 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.271966    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:15:40.273978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.273946    4453 cpu_manager.go:414] "containerMap: removing container" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:40.273978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.273977    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:40.274163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274005    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:40.274163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274013    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:40.274229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274206    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c40408-a994-4896-9b32-4239a64be60a" containerName="cleanup"
Oct 30 12:15:40.274229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274219    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d569e1-92ac-48ee-a6c8-ba73e66939ad" containerName="cleanup"
Oct 30 12:15:40.274295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274235    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d265ee56-f9d1-4551-8e31-5dddc2cdb362" containerName="cleanup"
Oct 30 12:15:40.274295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274246    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7500c47b-c7fc-42e9-8ba8-daf04b6b0157" containerName="cleanup"
Oct 30 12:15:40.274295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.274257    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b073803c-ec84-4dcf-a7a0-6155693811c5" containerName="cleanup"
Oct 30 12:15:40.280039 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.280012    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.294924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.294893    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:15:40.295496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.295475    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b15ec3-dockercfg-qphq5\""
Oct 30 12:15:40.334033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.334000    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b15ec3\""
Oct 30 12:15:40.392354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.388781    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.392354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.388897    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwhd\" (UniqueName: \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.490498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.490416    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.490647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.490533    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwhd\" (UniqueName: \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.491270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.491247    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.510867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.510839    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwhd\" (UniqueName: \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\") pod \"payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.596092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.596058    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:40.905698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:40.905663    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:15:40.908393 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:40.908359    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba0dbf8_9300_41c3_867d_81aae4ccda53.slice/crio-1a36f4a2f96f96308035b8be3d15e1184fd8d4f05dbce99c8c85826b61e140e8 WatchSource:0}: Error finding container 1a36f4a2f96f96308035b8be3d15e1184fd8d4f05dbce99c8c85826b61e140e8: Status 404 returned error can't find the container with id 1a36f4a2f96f96308035b8be3d15e1184fd8d4f05dbce99c8c85826b61e140e8
Oct 30 12:15:41.257896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:41.257794    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" event={"ID":"7ba0dbf8-9300-41c3-867d-81aae4ccda53","Type":"ContainerStarted","Data":"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"}
Oct 30 12:15:41.257896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:41.257867    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" event={"ID":"7ba0dbf8-9300-41c3-867d-81aae4ccda53","Type":"ContainerStarted","Data":"1a36f4a2f96f96308035b8be3d15e1184fd8d4f05dbce99c8c85826b61e140e8"}
Oct 30 12:15:41.258371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:41.257989    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:41.296406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:41.296346    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" podStartSLOduration=1.296312324 podStartE2EDuration="1.296312324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:41.29213013 +0000 UTC m=+2740.512415039" watchObservedRunningTime="2026-10-30 12:15:41.296312324 +0000 UTC m=+2740.516597236"
Oct 30 12:15:42.567562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.567512    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" podUID="8d60c83c-e724-4002-8597-ad6124a6de0f" containerName="payload-processing" containerID="cri-o://440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8" gracePeriod=25
Oct 30 12:15:42.841985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.841954    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:15:42.922674 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.922643    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\") pod \"8d60c83c-e724-4002-8597-ad6124a6de0f\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") "
Oct 30 12:15:42.922874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.922695    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\") pod \"8d60c83c-e724-4002-8597-ad6124a6de0f\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") "
Oct 30 12:15:42.922874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.922724    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\") pod \"8d60c83c-e724-4002-8597-ad6124a6de0f\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") "
Oct 30 12:15:42.923149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.923125    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\") pod \"8d60c83c-e724-4002-8597-ad6124a6de0f\" (UID: \"8d60c83c-e724-4002-8597-ad6124a6de0f\") "
Oct 30 12:15:42.931817 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.931784    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume" pod "8d60c83c-e724-4002-8597-ad6124a6de0f" (UID: "8d60c83c-e724-4002-8597-ad6124a6de0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:42.932284 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.932259    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume" pod "8d60c83c-e724-4002-8597-ad6124a6de0f" (UID: "8d60c83c-e724-4002-8597-ad6124a6de0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:15:42.937095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.937071    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2" pod "8d60c83c-e724-4002-8597-ad6124a6de0f" (UID: "8d60c83c-e724-4002-8597-ad6124a6de0f"). InnerVolumeSpecName "kube-api-access-db8s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:42.937174 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:42.937105    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials" pod "8d60c83c-e724-4002-8597-ad6124a6de0f" (UID: "8d60c83c-e724-4002-8597-ad6124a6de0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:15:43.025026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.024991    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:43.025026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.025018    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:43.025026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.025029    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d60c83c-e724-4002-8597-ad6124a6de0f-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:43.025245 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.025043    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8s2\" (UniqueName: \"kubernetes.io/projected/8d60c83c-e724-4002-8597-ad6124a6de0f-kube-api-access-db8s2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:15:43.279910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.279872    4453 generic.go:356] "Generic (PLEG): container finished" podID="8d60c83c-e724-4002-8597-ad6124a6de0f" containerID="440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8" exitCode=0
Oct 30 12:15:43.280097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.279924    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"
Oct 30 12:15:43.280097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.279962    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" event={"ID":"8d60c83c-e724-4002-8597-ad6124a6de0f","Type":"ContainerDied","Data":"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"}
Oct 30 12:15:43.280097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.280039    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf" event={"ID":"8d60c83c-e724-4002-8597-ad6124a6de0f","Type":"ContainerDied","Data":"26e4bd2a733c8090a010f28d16d7361eddfefda5abd7d8793d48ffc388f007cf"}
Oct 30 12:15:43.280097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.280067    4453 scope.go:122] "RemoveContainer" containerID="440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"
Oct 30 12:15:43.290732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.290711    4453 scope.go:122] "RemoveContainer" containerID="440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"
Oct 30 12:15:43.291004 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:15:43.290985    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8\": container with ID starting with 440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8 not found: ID does not exist" containerID="440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"
Oct 30 12:15:43.291056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.291019    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8"} err="failed to get container status \"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8\": rpc error: code = NotFound desc = could not find container \"440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8\": container with ID starting with 440c4ca191b1ebbfc2be5e34eb94859cea86b81b0b9466ef70fc57842c88efd8 not found: ID does not exist"
Oct 30 12:15:43.323874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.323852    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:15:43.332754 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:43.332730    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846099/payload-processing-external-model-e2e-extmod-b-846099-bc85b8lbf"]
Oct 30 12:15:44.912629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:44.912552    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d60c83c-e724-4002-8597-ad6124a6de0f" path="/var/lib/kubelet/pods/8d60c83c-e724-4002-8597-ad6124a6de0f/volumes"
Oct 30 12:15:52.268950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:52.268916    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:15:53.821751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.821717    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:15:53.822427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.822411    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8d60c83c-e724-4002-8597-ad6124a6de0f" containerName="payload-processing"
Oct 30 12:15:53.822485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.822428    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d60c83c-e724-4002-8597-ad6124a6de0f" containerName="payload-processing"
Oct 30 12:15:53.822521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.822512    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d60c83c-e724-4002-8597-ad6124a6de0f" containerName="payload-processing"
Oct 30 12:15:53.825599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.825582    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.837951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.837929    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"routing-overlay\""
Oct 30 12:15:53.847276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.847242    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"openshift-service-ca.crt\""
Oct 30 12:15:53.847276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.847245    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4f7443\""
Oct 30 12:15:53.847498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.847288    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"kube-root-ca.crt\""
Oct 30 12:15:53.847498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.847484    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-097f9a689008\""
Oct 30 12:15:53.847598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.847579    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-e2e--9f9529d9-dockercfg-fdxmq\""
Oct 30 12:15:53.850710 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.850690    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:15:53.872700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.872659    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.872852 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.872785    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.873005 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.872983    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.873080 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.873058    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf225\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.974855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.974805    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.975060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.974964    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf225\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.975060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.975053    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.975180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.975158    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.976146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.976108    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.976498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.976469    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.979158 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.979132    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:53.996131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:53.996098    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf225\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:54.138952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:54.138919    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:54.440559 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:54.440528    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:15:54.441966 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:15:54.441936    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5333a8f_bb9a_43f5_855c_1880a34bde40.slice/crio-aa192576da6e84929b327895bb385bb1a0a1f98c7e628ea56d7898b286a9ccf4 WatchSource:0}: Error finding container aa192576da6e84929b327895bb385bb1a0a1f98c7e628ea56d7898b286a9ccf4: Status 404 returned error can't find the container with id aa192576da6e84929b327895bb385bb1a0a1f98c7e628ea56d7898b286a9ccf4
Oct 30 12:15:55.382373 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:55.382305    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" event={"ID":"c5333a8f-bb9a-43f5-855c-1880a34bde40","Type":"ContainerStarted","Data":"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"}
Oct 30 12:15:55.382872 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:55.382400    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" event={"ID":"c5333a8f-bb9a-43f5-855c-1880a34bde40","Type":"ContainerStarted","Data":"aa192576da6e84929b327895bb385bb1a0a1f98c7e628ea56d7898b286a9ccf4"}
Oct 30 12:15:55.382872 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:55.382642    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:15:55.418504 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:15:55.418456    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" podStartSLOduration=2.418443893 podStartE2EDuration="2.418443893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:15:55.418160585 +0000 UTC m=+2754.638445494" watchObservedRunningTime="2026-10-30 12:15:55.418443893 +0000 UTC m=+2754.638728803"
Oct 30 12:16:03.686058 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.686025    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:16:03.690140 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.690115    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.702166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.702140    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-67e819-serving-cert\""
Oct 30 12:16:03.716423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.716382    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:16:03.790731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.790696    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.790894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.790871    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.790934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.790913    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.791063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.791049    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nng6v\" (UniqueName: \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.893356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.892064    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.893356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.892145    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.893356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.892292    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nng6v\" (UniqueName: \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.893356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.892370    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.893759 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.893499    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.898898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.898860    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.900267 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.900139    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:03.914213 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:03.914182    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nng6v\" (UniqueName: \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\") pod \"maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:04.002829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:04.002747    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:04.307029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:04.306999    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:16:04.308029 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:16:04.308002    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bbce70_cf88_4c88_8952_ec2f3797e382.slice/crio-c38c17c78884dbfec0c6268c3eae4feafc05151f40654945819a69a9f571bbff WatchSource:0}: Error finding container c38c17c78884dbfec0c6268c3eae4feafc05151f40654945819a69a9f571bbff: Status 404 returned error can't find the container with id c38c17c78884dbfec0c6268c3eae4feafc05151f40654945819a69a9f571bbff
Oct 30 12:16:04.438845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:04.438809    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerStarted","Data":"c38c17c78884dbfec0c6268c3eae4feafc05151f40654945819a69a9f571bbff"}
Oct 30 12:16:05.447191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:05.446606    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerStarted","Data":"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"}
Oct 30 12:16:05.447191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:05.446681    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:05.485118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:05.485061    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" podStartSLOduration=2.092629405 podStartE2EDuration="2.485043583s" totalImagesPullingTime="392.414178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:05.482866395 +0000 UTC m=+2764.703151305" watchObservedRunningTime="2026-10-30 12:16:05.485043583 +0000 UTC m=+2764.705328494"
Oct 30 12:16:06.391123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:06.391092    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:16:07.570830 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:16:07.570795    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:16:09.729081 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:09.729049    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 12:16:09.729559 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:09.729258    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" containerID="cri-o://0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f" gracePeriod=30
Oct 30 12:16:11.387791 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:11.387760    4453 patch_prober.go:29] interesting pod/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:16:11.388260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:11.387818    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:16:11.454820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:11.454785    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.128:8443/health\": dial tcp 10.133.0.128:8443: connect: connection refused"
Oct 30 12:16:14.989238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:14.989211    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 12:16:15.044341 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044252    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044341 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044313    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044540 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044396    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044540 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044467    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044540 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044491    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044593    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044624    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044657    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044715    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\") pod \"24f3860b-0568-42a4-aba9-bc0876ef24b5\" (UID: \"24f3860b-0568-42a4-aba9-bc0876ef24b5\") "
Oct 30 12:16:15.044879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044589    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:16:15.044879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044778    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:16:15.045035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.044982    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:16:15.045200 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.045182    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.045282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.045201    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.045282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.045212    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.045705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.045640    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:16:15.046199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.046147    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:16:15.047079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.047052    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "kube-api-access-tdh54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:16:15.047241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.047217    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:16:15.047241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.047233    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:16:15.047729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.047701    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy" pod "24f3860b-0568-42a4-aba9-bc0876ef24b5" (UID: "24f3860b-0568-42a4-aba9-bc0876ef24b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:16:15.146230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146199    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.146230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146227    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24f3860b-0568-42a4-aba9-bc0876ef24b5-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.146230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146235    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.146552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146243    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdh54\" (UniqueName: \"kubernetes.io/projected/24f3860b-0568-42a4-aba9-bc0876ef24b5-kube-api-access-tdh54\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.146552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146253    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.146552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.146262    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24f3860b-0568-42a4-aba9-bc0876ef24b5-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:16:15.528778 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.528744    4453 generic.go:356] "Generic (PLEG): container finished" podID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerID="834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52" exitCode=1
Oct 30 12:16:15.528969 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.528816    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerDied","Data":"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"}
Oct 30 12:16:15.529320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.529293    4453 scope.go:122] "RemoveContainer" containerID="834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"
Oct 30 12:16:15.530226 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.530199    4453 generic.go:356] "Generic (PLEG): container finished" podID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerID="0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f" exitCode=0
Oct 30 12:16:15.530349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.530262    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"
Oct 30 12:16:15.530349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.530300    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" event={"ID":"24f3860b-0568-42a4-aba9-bc0876ef24b5","Type":"ContainerDied","Data":"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"}
Oct 30 12:16:15.530496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.530388    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw" event={"ID":"24f3860b-0568-42a4-aba9-bc0876ef24b5","Type":"ContainerDied","Data":"1a9f996b1f953029c25903fed848df0b6e93fe14e8efaa9de84f378243cadc67"}
Oct 30 12:16:15.530496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.530404    4453 scope.go:122] "RemoveContainer" containerID="0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"
Oct 30 12:16:15.540271 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.540250    4453 scope.go:122] "RemoveContainer" containerID="0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"
Oct 30 12:16:15.540548 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:16:15.540528    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f\": container with ID starting with 0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f not found: ID does not exist" containerID="0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"
Oct 30 12:16:15.540595 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.540555    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f"} err="failed to get container status \"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f\": rpc error: code = NotFound desc = could not find container \"0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f\": container with ID starting with 0e5897a067b0dab69cbb3a6627d53b3dc1457e2c6969df9c35716a9e8dc1130f not found: ID does not exist"
Oct 30 12:16:15.599110 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.599079    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 12:16:15.602655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.602629    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-mw8fw"]
Oct 30 12:16:15.853655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.853574    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:16:15.854232 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.854217    4453 cpu_manager.go:414] "containerMap: removing container" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy"
Oct 30 12:16:15.854291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.854232    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy"
Oct 30 12:16:15.854354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.854344    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" containerName="istio-proxy"
Oct 30 12:16:15.857531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.857511    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:15.871168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.871145    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b15ec3-serving-cert\""
Oct 30 12:16:15.879460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.879435    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:16:15.954873 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.954838    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:15.955058 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.954889    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k725p\" (UniqueName: \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:15.955150 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.955131    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:15.955199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:15.955186    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.056188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.056155    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.056705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.056203    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.056705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.056421    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.056705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.056475    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k725p\" (UniqueName: \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.057012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.056992    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.058880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.058854    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.058993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.058953    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.083569 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.083019    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k725p\" (UniqueName: \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\") pod \"maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.171054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.171017    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:16.475863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.475826    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:16:16.476911 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:16:16.476886    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf19fa44_557b_4b16_9d34_e39ecaf9ac08.slice/crio-3e2c89a9a3e95616397f4a62d7b6114583e343aea97ff38131d35c16551d8939 WatchSource:0}: Error finding container 3e2c89a9a3e95616397f4a62d7b6114583e343aea97ff38131d35c16551d8939: Status 404 returned error can't find the container with id 3e2c89a9a3e95616397f4a62d7b6114583e343aea97ff38131d35c16551d8939
Oct 30 12:16:16.537853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.537823    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" event={"ID":"af19fa44-557b-4b16-9d34-e39ecaf9ac08","Type":"ContainerStarted","Data":"3e2c89a9a3e95616397f4a62d7b6114583e343aea97ff38131d35c16551d8939"}
Oct 30 12:16:16.539830 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.539795    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerStarted","Data":"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"}
Oct 30 12:16:16.540008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.539990    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:16.924650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:16.924613    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f3860b-0568-42a4-aba9-bc0876ef24b5" path="/var/lib/kubelet/pods/24f3860b-0568-42a4-aba9-bc0876ef24b5/volumes"
Oct 30 12:16:17.548838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:17.548789    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" event={"ID":"af19fa44-557b-4b16-9d34-e39ecaf9ac08","Type":"ContainerStarted","Data":"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"}
Oct 30 12:16:17.549313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:17.549001    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:17.589434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:17.589379    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" podStartSLOduration=2.251599503 podStartE2EDuration="2.589365396s" totalImagesPullingTime="337.765893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:17.585925549 +0000 UTC m=+2776.806210455" watchObservedRunningTime="2026-10-30 12:16:17.589365396 +0000 UTC m=+2776.809650306"
Oct 30 12:16:22.554921 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:22.554892    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:16:23.562105 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:23.562075    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:16:37.592521 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:16:37.592449    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:16:57.959644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.959611    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:16:57.964188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.964169    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:57.976555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.976532    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-e7b0ccb0b1c8\""
Oct 30 12:16:57.985945 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.985922    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-6fe65b203240\""
Oct 30 12:16:57.986096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.985968    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-e2e--b2e93605-dockercfg-8bqs8\""
Oct 30 12:16:57.986096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.986008    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"agc-cred-16c43061b500\""
Oct 30 12:16:57.986217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.986022    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"routing-overlay\""
Oct 30 12:16:57.986217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.986055    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"kube-root-ca.crt\""
Oct 30 12:16:57.986217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.986060    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"openshift-service-ca.crt\""
Oct 30 12:16:57.986217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.986148    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b8a9d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7b8a9d\""
Oct 30 12:16:57.991100 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:57.991077    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:16:58.008274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.008244    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.008489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.008470    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.008561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.008519    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.008617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.008598    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jvk\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.109875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.109842    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.110023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.109952    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.110023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.109978    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.110099 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.110019    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jvk\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.110606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.110582    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.110816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.110794    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.112767 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.112742    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.130899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.130879    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jvk\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.278871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.278781    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.580871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.580842    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:16:58.581974 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:16:58.581946    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32ae31b_e0e6_430d_89d0_f79c899af1d2.slice/crio-2572adb7550722edd0abc86d5c2306a7e7541fafb1aba54eae722a2c270ea76c WatchSource:0}: Error finding container 2572adb7550722edd0abc86d5c2306a7e7541fafb1aba54eae722a2c270ea76c: Status 404 returned error can't find the container with id 2572adb7550722edd0abc86d5c2306a7e7541fafb1aba54eae722a2c270ea76c
Oct 30 12:16:58.820318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.820285    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" event={"ID":"b32ae31b-e0e6-430d-89d0-f79c899af1d2","Type":"ContainerStarted","Data":"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"}
Oct 30 12:16:58.820508 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.820349    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" event={"ID":"b32ae31b-e0e6-430d-89d0-f79c899af1d2","Type":"ContainerStarted","Data":"2572adb7550722edd0abc86d5c2306a7e7541fafb1aba54eae722a2c270ea76c"}
Oct 30 12:16:58.820508 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.820387    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:16:58.857475 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:16:58.857423    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" podStartSLOduration=1.857410524 podStartE2EDuration="1.857410524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:16:58.856827554 +0000 UTC m=+2818.077112464" watchObservedRunningTime="2026-10-30 12:16:58.857410524 +0000 UTC m=+2818.077695434"
Oct 30 12:17:05.938232 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:05.938196    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:17:05.943655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:05.943637    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:05.968599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:05.968573    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bae155\""
Oct 30 12:17:05.968599 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:05.968589    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bae155-dockercfg-7lgbr\""
Oct 30 12:17:05.973245 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:05.973223    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:17:06.040372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.040063    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.040372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.040265    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh7h\" (UniqueName: \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.090864 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.090802    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:17:06.097591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.097559    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.122401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.122356    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:17:06.141728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.141693    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.141957 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.141935    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh7h\" (UniqueName: \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.142663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.142637    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.164620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.164585    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh7h\" (UniqueName: \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\") pod \"payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.248182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.248063    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.248406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.248204    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh928\" (UniqueName: \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.260167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.259022    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.349763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.349732    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.349928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.349818    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh928\" (UniqueName: \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.350652 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.350622    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.375800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.375220    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh928\" (UniqueName: \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\") pod \"payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.399123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.399088    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:17:06.421571 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.421537    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.606115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.606081    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:17:06.607588 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:17:06.607555    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef703c0_80d5_43e8_b458_2e71c4532e84.slice/crio-a2199a89f8aecfd58de93b51d0279cb1676326a5fc2a13f4c619fbb3b077b0e8 WatchSource:0}: Error finding container a2199a89f8aecfd58de93b51d0279cb1676326a5fc2a13f4c619fbb3b077b0e8: Status 404 returned error can't find the container with id a2199a89f8aecfd58de93b51d0279cb1676326a5fc2a13f4c619fbb3b077b0e8
Oct 30 12:17:06.737536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.737505    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:17:06.738189 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:17:06.738156    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6cc14_c200_4493_97a2_92b5d7f70139.slice/crio-e00f572374498eabd8b82a4186ab0225c1df17723bc036febe181f470877c923 WatchSource:0}: Error finding container e00f572374498eabd8b82a4186ab0225c1df17723bc036febe181f470877c923: Status 404 returned error can't find the container with id e00f572374498eabd8b82a4186ab0225c1df17723bc036febe181f470877c923
Oct 30 12:17:06.878423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.878389    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" event={"ID":"23c6cc14-c200-4493-97a2-92b5d7f70139","Type":"ContainerStarted","Data":"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"}
Oct 30 12:17:06.878555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.878451    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" event={"ID":"23c6cc14-c200-4493-97a2-92b5d7f70139","Type":"ContainerStarted","Data":"e00f572374498eabd8b82a4186ab0225c1df17723bc036febe181f470877c923"}
Oct 30 12:17:06.878638 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.878603    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:06.880409 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.880380    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" event={"ID":"def703c0-80d5-43e8-b458-2e71c4532e84","Type":"ContainerStarted","Data":"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"}
Oct 30 12:17:06.880529 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.880429    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" event={"ID":"def703c0-80d5-43e8-b458-2e71c4532e84","Type":"ContainerStarted","Data":"a2199a89f8aecfd58de93b51d0279cb1676326a5fc2a13f4c619fbb3b077b0e8"}
Oct 30 12:17:06.880529 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.880512    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:06.915478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.915432    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" podStartSLOduration=0.915419897 podStartE2EDuration="915.419897ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:06.913790281 +0000 UTC m=+2826.134075172" watchObservedRunningTime="2026-10-30 12:17:06.915419897 +0000 UTC m=+2826.135704806"
Oct 30 12:17:06.951645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:06.951600    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" podStartSLOduration=1.951585069 podStartE2EDuration="1.951585069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:06.94878437 +0000 UTC m=+2826.169069280" watchObservedRunningTime="2026-10-30 12:17:06.951585069 +0000 UTC m=+2826.171870018"
Oct 30 12:17:07.572002 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:17:07.571969    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:17:09.831488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:09.831461    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:17:11.400718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.400638    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" podUID="b96ce321-2f6f-4528-af22-e14957d333c1" containerName="payload-processing" containerID="cri-o://3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705" gracePeriod=25
Oct 30 12:17:11.736561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.736535    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:17:11.823997 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.823958    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\") pod \"b96ce321-2f6f-4528-af22-e14957d333c1\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") "
Oct 30 12:17:11.824182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824065    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\") pod \"b96ce321-2f6f-4528-af22-e14957d333c1\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") "
Oct 30 12:17:11.824182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824152    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\") pod \"b96ce321-2f6f-4528-af22-e14957d333c1\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") "
Oct 30 12:17:11.824299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824228    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\") pod \"b96ce321-2f6f-4528-af22-e14957d333c1\" (UID: \"b96ce321-2f6f-4528-af22-e14957d333c1\") "
Oct 30 12:17:11.824552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824523    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume" pod "b96ce321-2f6f-4528-af22-e14957d333c1" (UID: "b96ce321-2f6f-4528-af22-e14957d333c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:11.824625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824543    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume" pod "b96ce321-2f6f-4528-af22-e14957d333c1" (UID: "b96ce321-2f6f-4528-af22-e14957d333c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:17:11.824741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824725    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:17:11.824799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.824745    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96ce321-2f6f-4528-af22-e14957d333c1-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:17:11.826656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.826626    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff" pod "b96ce321-2f6f-4528-af22-e14957d333c1" (UID: "b96ce321-2f6f-4528-af22-e14957d333c1"). InnerVolumeSpecName "kube-api-access-q7dff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:11.826800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.826733    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials" pod "b96ce321-2f6f-4528-af22-e14957d333c1" (UID: "b96ce321-2f6f-4528-af22-e14957d333c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:17:11.924051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.924014    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:17:11.924722 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.924695    4453 generic.go:356] "Generic (PLEG): container finished" podID="b96ce321-2f6f-4528-af22-e14957d333c1" containerID="3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705" exitCode=0
Oct 30 12:17:11.924888 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.924763    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b96ce321-2f6f-4528-af22-e14957d333c1" containerName="payload-processing"
Oct 30 12:17:11.924888 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.924779    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96ce321-2f6f-4528-af22-e14957d333c1" containerName="payload-processing"
Oct 30 12:17:11.924888 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.924806    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"
Oct 30 12:17:11.925087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.925074    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96ce321-2f6f-4528-af22-e14957d333c1" containerName="payload-processing"
Oct 30 12:17:11.925605 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.925583    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:17:11.925703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.925607    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dff\" (UniqueName: \"kubernetes.io/projected/b96ce321-2f6f-4528-af22-e14957d333c1-kube-api-access-q7dff\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:17:11.928929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.928907    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" event={"ID":"b96ce321-2f6f-4528-af22-e14957d333c1","Type":"ContainerDied","Data":"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"}
Oct 30 12:17:11.929027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.928947    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82" event={"ID":"b96ce321-2f6f-4528-af22-e14957d333c1","Type":"ContainerDied","Data":"b8d1657461e9683977017d60affce7e3b88d6ec67d5b5d887aa25a6a6d7891b0"}
Oct 30 12:17:11.929078 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.929042    4453 scope.go:122] "RemoveContainer" containerID="3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"
Oct 30 12:17:11.929145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.929125    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:11.939244 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.939222    4453 scope.go:122] "RemoveContainer" containerID="3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"
Oct 30 12:17:11.939713 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:17:11.939623    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705\": container with ID starting with 3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705 not found: ID does not exist" containerID="3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"
Oct 30 12:17:11.939713 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.939660    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705"} err="failed to get container status \"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705\": rpc error: code = NotFound desc = could not find container \"3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705\": container with ID starting with 3e643c2fc6b82d9b7c8c0ff68322c0037351d424afe23aa35dad2c9346f14705 not found: ID does not exist"
Oct 30 12:17:11.944426 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.944405    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f8f132\""
Oct 30 12:17:11.944546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.944454    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-qmscv\""
Oct 30 12:17:11.945160 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.945144    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"routing-overlay\""
Oct 30 12:17:11.948364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.948315    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:17:11.995902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.995818    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"kube-root-ca.crt\""
Oct 30 12:17:11.996072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:11.996030    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"openshift-service-ca.crt\""
Oct 30 12:17:12.007046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.007017    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:17:12.010221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.010196    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-5f49kcs82"]
Oct 30 12:17:12.026495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.026469    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.026612 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.026575    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.026667 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.026654    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.026793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.026780    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8p8\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.045180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.045161    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"e2e-cfl-58cf8a-provider-cred\""
Oct 30 12:17:12.127466 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.127429    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.127683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.127549    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8p8\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.127683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.127591    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.127683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.127646    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.128254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.128228    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.128375 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.128296    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.130036 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.130017    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.148220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.148195    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8p8\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.247882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.247796    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.553369 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.553319    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:17:12.554517 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:17:12.554487    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbcb89c_5c6f_4769_bee8_1265fbf7e19f.slice/crio-c1d5e24ce7cb892e70d44dce1398928a014424eabbca781352f051ac8c020da1 WatchSource:0}: Error finding container c1d5e24ce7cb892e70d44dce1398928a014424eabbca781352f051ac8c020da1: Status 404 returned error can't find the container with id c1d5e24ce7cb892e70d44dce1398928a014424eabbca781352f051ac8c020da1
Oct 30 12:17:12.912843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.912809    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96ce321-2f6f-4528-af22-e14957d333c1" path="/var/lib/kubelet/pods/b96ce321-2f6f-4528-af22-e14957d333c1/volumes"
Oct 30 12:17:12.932609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.932574    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" event={"ID":"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f","Type":"ContainerStarted","Data":"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"}
Oct 30 12:17:12.932792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.932630    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" event={"ID":"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f","Type":"ContainerStarted","Data":"c1d5e24ce7cb892e70d44dce1398928a014424eabbca781352f051ac8c020da1"}
Oct 30 12:17:12.932792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.932746    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:12.968028 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:12.967975    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" podStartSLOduration=1.9679606349999998 podStartE2EDuration="1.967960635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:12.967743391 +0000 UTC m=+2832.188028302" watchObservedRunningTime="2026-10-30 12:17:12.967960635 +0000 UTC m=+2832.188245545"
Oct 30 12:17:17.899159 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:17.899079    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:17:17.899561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:17.899160    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:17:23.943822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:23.943787    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:17:37.573911 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:17:37.573869    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:17:45.992398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:45.992361    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:17:45.997200 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:45.997179    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.010227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.010207    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bae155-serving-cert\""
Oct 30 12:17:46.023793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.023770    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:17:46.069107 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.069070    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.069269 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.069245    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.069469 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.069448    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkctt\" (UniqueName: \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.069584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.069564    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.170664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.170633    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.170883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.170753    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkctt\" (UniqueName: \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.170883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.170820    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.171006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.170932    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.171476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.171445    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.173776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.173749    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.173891 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.173792    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.192372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.192344    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkctt\" (UniqueName: \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\") pod \"maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.311251 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.311169    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:46.613811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.613785    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:17:46.615256 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:17:46.615221    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98444241_0a51_467c_9446_16485e10d06f.slice/crio-67136245ab981bb6a6af65a506c87b51b72de633dae1963324e5f76565b606af WatchSource:0}: Error finding container 67136245ab981bb6a6af65a506c87b51b72de633dae1963324e5f76565b606af: Status 404 returned error can't find the container with id 67136245ab981bb6a6af65a506c87b51b72de633dae1963324e5f76565b606af
Oct 30 12:17:46.616613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:46.616594    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:17:47.151245 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:47.151210    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" event={"ID":"98444241-0a51-467c-9446-16485e10d06f","Type":"ContainerStarted","Data":"665a7b1bebab8156e252dc922c793174556063ebfa6518626092670f083690e7"}
Oct 30 12:17:47.151741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:47.151270    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" event={"ID":"98444241-0a51-467c-9446-16485e10d06f","Type":"ContainerStarted","Data":"67136245ab981bb6a6af65a506c87b51b72de633dae1963324e5f76565b606af"}
Oct 30 12:17:47.152069 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:47.152036    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:17:47.195244 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:47.195183    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" podStartSLOduration=1.849330938 podStartE2EDuration="2.195165002s" totalImagesPullingTime="345.834064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:17:47.194601869 +0000 UTC m=+2866.414886780" watchObservedRunningTime="2026-10-30 12:17:47.195165002 +0000 UTC m=+2866.415449918"
Oct 30 12:17:53.162697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:17:53.162664    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:18:06.000046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:06.000011    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:18:07.578888 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:07.578848    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:18:11.000951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.000875    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" podUID="c5333a8f-bb9a-43f5-855c-1880a34bde40" containerName="payload-processing" containerID="cri-o://97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d" gracePeriod=25
Oct 30 12:18:11.302098 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.302068    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:18:11.308720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.308691    4453 generic.go:356] "Generic (PLEG): container finished" podID="c5333a8f-bb9a-43f5-855c-1880a34bde40" containerID="97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d" exitCode=0
Oct 30 12:18:11.308854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.308729    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"
Oct 30 12:18:11.308854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.308771    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" event={"ID":"c5333a8f-bb9a-43f5-855c-1880a34bde40","Type":"ContainerDied","Data":"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"}
Oct 30 12:18:11.308854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.308832    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562" event={"ID":"c5333a8f-bb9a-43f5-855c-1880a34bde40","Type":"ContainerDied","Data":"aa192576da6e84929b327895bb385bb1a0a1f98c7e628ea56d7898b286a9ccf4"}
Oct 30 12:18:11.308960 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.308859    4453 scope.go:122] "RemoveContainer" containerID="97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"
Oct 30 12:18:11.318388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.318369    4453 scope.go:122] "RemoveContainer" containerID="97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"
Oct 30 12:18:11.318674 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:11.318650    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d\": container with ID starting with 97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d not found: ID does not exist" containerID="97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"
Oct 30 12:18:11.318731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.318685    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d"} err="failed to get container status \"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d\": rpc error: code = NotFound desc = could not find container \"97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d\": container with ID starting with 97ba917b35e0379cf7da8227b37a62344073dd3415e1b8e895aec5d4acfcce1d not found: ID does not exist"
Oct 30 12:18:11.391123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391086    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\") pod \"c5333a8f-bb9a-43f5-855c-1880a34bde40\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") "
Oct 30 12:18:11.391346 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391147    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\") pod \"c5333a8f-bb9a-43f5-855c-1880a34bde40\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") "
Oct 30 12:18:11.391346 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391207    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\") pod \"c5333a8f-bb9a-43f5-855c-1880a34bde40\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") "
Oct 30 12:18:11.391346 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391313    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\") pod \"c5333a8f-bb9a-43f5-855c-1880a34bde40\" (UID: \"c5333a8f-bb9a-43f5-855c-1880a34bde40\") "
Oct 30 12:18:11.391604 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391579    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume" pod "c5333a8f-bb9a-43f5-855c-1880a34bde40" (UID: "c5333a8f-bb9a-43f5-855c-1880a34bde40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:11.391669 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391646    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume" pod "c5333a8f-bb9a-43f5-855c-1880a34bde40" (UID: "c5333a8f-bb9a-43f5-855c-1880a34bde40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:11.391819 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391802    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:11.391855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.391824    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5333a8f-bb9a-43f5-855c-1880a34bde40-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:11.393625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.393603    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225" pod "c5333a8f-bb9a-43f5-855c-1880a34bde40" (UID: "c5333a8f-bb9a-43f5-855c-1880a34bde40"). InnerVolumeSpecName "kube-api-access-sf225". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:11.393688 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.393639    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials" pod "c5333a8f-bb9a-43f5-855c-1880a34bde40" (UID: "c5333a8f-bb9a-43f5-855c-1880a34bde40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:11.493531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.493491    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:11.493531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.493523    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf225\" (UniqueName: \"kubernetes.io/projected/c5333a8f-bb9a-43f5-855c-1880a34bde40-kube-api-access-sf225\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:11.649827 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.649785    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:18:11.659073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:11.659042    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f44z562"]
Oct 30 12:18:12.914564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:12.914523    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5333a8f-bb9a-43f5-855c-1880a34bde40" path="/var/lib/kubelet/pods/c5333a8f-bb9a-43f5-855c-1880a34bde40/volumes"
Oct 30 12:18:24.977419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:24.977374    4453 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-18.ec2.internal" podUID="6905fd60ff57bade2ab9f0792ffa70c2" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:18:34.670905 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.670868    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:34.671659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.671643    4453 cpu_manager.go:414] "containerMap: removing container" podUID="c5333a8f-bb9a-43f5-855c-1880a34bde40" containerName="payload-processing"
Oct 30 12:18:34.671659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.671660    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5333a8f-bb9a-43f5-855c-1880a34bde40" containerName="payload-processing"
Oct 30 12:18:34.671772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.671762    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5333a8f-bb9a-43f5-855c-1880a34bde40" containerName="payload-processing"
Oct 30 12:18:34.675433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.675408    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.687901 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.687872    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"agc-cred-5104d21e3d4d\""
Oct 30 12:18:34.697315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697271    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"openshift-service-ca.crt\""
Oct 30 12:18:34.697315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697302    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"agc-cred-1b7cc153caa8\""
Oct 30 12:18:34.697514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697302    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"kube-root-ca.crt\""
Oct 30 12:18:34.697514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697301    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b15ec3\""
Oct 30 12:18:34.697514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697289    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"routing-overlay\""
Oct 30 12:18:34.697718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.697696    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-e2e--a1d927ff-dockercfg-lkfzd\""
Oct 30 12:18:34.702159 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.702136    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:34.802742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.802706    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.802941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.802881    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.802941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.802927    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.803109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.803093    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.904598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.904557    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.904772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.904701    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.904772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.904745    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.904907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.904883    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.905307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.905284    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.905518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.905498    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.907344 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.907311    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.926190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.926135    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:34.992072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:34.992042    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:35.294995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:35.294960    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:35.296967 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:18:35.296939    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea1da20_c042_4f34_b411_55e677f76a9f.slice/crio-9bb21128df36d2512d8631c16dc112357b90f994aa29b47e912853ceca5acb3d WatchSource:0}: Error finding container 9bb21128df36d2512d8631c16dc112357b90f994aa29b47e912853ceca5acb3d: Status 404 returned error can't find the container with id 9bb21128df36d2512d8631c16dc112357b90f994aa29b47e912853ceca5acb3d
Oct 30 12:18:35.477055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:35.475917    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" event={"ID":"2ea1da20-c042-4f34-b411-55e677f76a9f","Type":"ContainerStarted","Data":"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"}
Oct 30 12:18:35.477055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:35.475982    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" event={"ID":"2ea1da20-c042-4f34-b411-55e677f76a9f","Type":"ContainerStarted","Data":"9bb21128df36d2512d8631c16dc112357b90f994aa29b47e912853ceca5acb3d"}
Oct 30 12:18:35.478237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:35.478206    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:35.514747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:35.514693    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" podStartSLOduration=1.5146793079999998 podStartE2EDuration="1.514679308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:35.513513474 +0000 UTC m=+2914.733798384" watchObservedRunningTime="2026-10-30 12:18:35.514679308 +0000 UTC m=+2914.734964218"
Oct 30 12:18:36.628730 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:36.628691    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b15ec3/agc-cred-5104d21e3d4d: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:36.628730 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:36.628724    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:36.629232 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:36.628800    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials podName:2ea1da20-c042-4f34-b411-55e677f76a9f nodeName:}" failed. No retries permitted until 2026-10-30 12:18:37.128780214 +0000 UTC m=+2916.349065115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f") : secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:37.136037 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:37.136010    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b15ec3/agc-cred-5104d21e3d4d: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:37.136037 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:37.136034    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:37.136245 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:37.136091    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials podName:2ea1da20-c042-4f34-b411-55e677f76a9f nodeName:}" failed. No retries permitted until 2026-10-30 12:18:38.1360755 +0000 UTC m=+2917.356360405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f") : secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:37.573551 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:37.573454    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:18:38.147315 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:38.147280    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b15ec3/agc-cred-5104d21e3d4d: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:38.147315 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:38.147305    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:38.147769 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:38.147381    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials podName:2ea1da20-c042-4f34-b411-55e677f76a9f nodeName:}" failed. No retries permitted until 2026-10-30 12:18:40.147366031 +0000 UTC m=+2919.367650923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f") : secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:38.704669 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:38.704634    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:40.179094 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:40.179057    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b15ec3/agc-cred-5104d21e3d4d: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:40.179094 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:40.179086    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7: secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:40.179671 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:40.179162    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials podName:2ea1da20-c042-4f34-b411-55e677f76a9f nodeName:}" failed. No retries permitted until 2026-10-30 12:18:44.179143469 +0000 UTC m=+2923.399428375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f") : secret "agc-cred-5104d21e3d4d" not found
Oct 30 12:18:43.705227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:43.705123    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" podUID="2ea1da20-c042-4f34-b411-55e677f76a9f" containerName="payload-processing" containerID="cri-o://1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2" gracePeriod=25
Oct 30 12:18:43.991802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:43.991775    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:44.048502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.048472    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\") pod \"2ea1da20-c042-4f34-b411-55e677f76a9f\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") "
Oct 30 12:18:44.048701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.048520    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\") pod \"2ea1da20-c042-4f34-b411-55e677f76a9f\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") "
Oct 30 12:18:44.048876 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.048842    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\") pod \"2ea1da20-c042-4f34-b411-55e677f76a9f\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") "
Oct 30 12:18:44.049048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.048878    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume" pod "2ea1da20-c042-4f34-b411-55e677f76a9f" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:44.049048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.048968    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\") pod \"2ea1da20-c042-4f34-b411-55e677f76a9f\" (UID: \"2ea1da20-c042-4f34-b411-55e677f76a9f\") "
Oct 30 12:18:44.049449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.049419    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume" pod "2ea1da20-c042-4f34-b411-55e677f76a9f" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:44.049682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.049660    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:44.049762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.049689    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea1da20-c042-4f34-b411-55e677f76a9f-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:44.051089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.051056    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials" pod "2ea1da20-c042-4f34-b411-55e677f76a9f" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:44.051416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.051390    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd" pod "2ea1da20-c042-4f34-b411-55e677f76a9f" (UID: "2ea1da20-c042-4f34-b411-55e677f76a9f"). InnerVolumeSpecName "kube-api-access-5lhxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:44.151237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.151204    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:44.151237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.151235    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhxd\" (UniqueName: \"kubernetes.io/projected/2ea1da20-c042-4f34-b411-55e677f76a9f-kube-api-access-5lhxd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:44.533188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.533150    4453 generic.go:356] "Generic (PLEG): container finished" podID="2ea1da20-c042-4f34-b411-55e677f76a9f" containerID="1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2" exitCode=0
Oct 30 12:18:44.533354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.533200    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"
Oct 30 12:18:44.533354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.533236    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" event={"ID":"2ea1da20-c042-4f34-b411-55e677f76a9f","Type":"ContainerDied","Data":"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"}
Oct 30 12:18:44.533354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.533294    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7" event={"ID":"2ea1da20-c042-4f34-b411-55e677f76a9f","Type":"ContainerDied","Data":"9bb21128df36d2512d8631c16dc112357b90f994aa29b47e912853ceca5acb3d"}
Oct 30 12:18:44.533354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.533316    4453 scope.go:122] "RemoveContainer" containerID="1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"
Oct 30 12:18:44.544525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.544508    4453 scope.go:122] "RemoveContainer" containerID="1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"
Oct 30 12:18:44.544823 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:44.544804    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2\": container with ID starting with 1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2 not found: ID does not exist" containerID="1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"
Oct 30 12:18:44.544863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.544832    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2"} err="failed to get container status \"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2\": rpc error: code = NotFound desc = could not find container \"1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2\": container with ID starting with 1012bd53d0cc3977cd53941e132fd7c74925279393e7cdd04ba9f0dd69985ef2 not found: ID does not exist"
Oct 30 12:18:44.578693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.578667    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:44.587299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.587278    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-74fbg5wv7"]
Oct 30 12:18:44.913139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:44.913105    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea1da20-c042-4f34-b411-55e677f76a9f" path="/var/lib/kubelet/pods/2ea1da20-c042-4f34-b411-55e677f76a9f/volumes"
Oct 30 12:18:50.787119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:50.787082    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:18:52.355352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.355292    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"]
Oct 30 12:18:52.355994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.355978    4453 cpu_manager.go:414] "containerMap: removing container" podUID="2ea1da20-c042-4f34-b411-55e677f76a9f" containerName="payload-processing"
Oct 30 12:18:52.356046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.355995    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea1da20-c042-4f34-b411-55e677f76a9f" containerName="payload-processing"
Oct 30 12:18:52.356106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.356097    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea1da20-c042-4f34-b411-55e677f76a9f" containerName="payload-processing"
Oct 30 12:18:52.359725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.359705    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.372583 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.372560    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-0c9e9832e52d\""
Oct 30 12:18:52.382138 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.382112    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-l26f8\""
Oct 30 12:18:52.386028 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.386003    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"]
Oct 30 12:18:52.476492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.476454    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlk92\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.484403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.476813    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.484699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.484673    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.484838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.484821    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.571117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.571077    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"]
Oct 30 12:18:52.571588 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:52.571556    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mlk92 plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs" podUID="bf9d8836-6fb6-459f-a05c-f95852bb9e73"
Oct 30 12:18:52.586508 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.586477    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlk92\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.586670 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.586514    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.586670 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.586562    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.586670 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.586588    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.586840 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:52.586671    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:52.586840 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:52.586692    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:52.586840 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:52.586747    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials podName:bf9d8836-6fb6-459f-a05c-f95852bb9e73 nodeName:}" failed. No retries permitted until 2026-10-30 12:18:53.086728226 +0000 UTC m=+2932.307013134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs" (UID: "bf9d8836-6fb6-459f-a05c-f95852bb9e73") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:52.587325 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.587304    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.587603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.587576    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.609832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.609803    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlk92\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.626898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.626863    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.640885 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.640860    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:52.788462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.788426    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\") pod \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") "
Oct 30 12:18:52.788672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.788486    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\") pod \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") "
Oct 30 12:18:52.788672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.788553    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\") pod \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") "
Oct 30 12:18:52.788937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.788910    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume" pod "bf9d8836-6fb6-459f-a05c-f95852bb9e73" (UID: "bf9d8836-6fb6-459f-a05c-f95852bb9e73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:52.789091 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.789001    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume" pod "bf9d8836-6fb6-459f-a05c-f95852bb9e73" (UID: "bf9d8836-6fb6-459f-a05c-f95852bb9e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:52.789241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.789223    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:52.789305 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.789247    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9d8836-6fb6-459f-a05c-f95852bb9e73-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:52.790674 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.790651    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92" pod "bf9d8836-6fb6-459f-a05c-f95852bb9e73" (UID: "bf9d8836-6fb6-459f-a05c-f95852bb9e73"). InnerVolumeSpecName "kube-api-access-mlk92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:52.890406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:52.890319    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlk92\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-kube-api-access-mlk92\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:53.093166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:53.093127    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs\" (UID: \"bf9d8836-6fb6-459f-a05c-f95852bb9e73\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:53.093368 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:53.093274    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-0c9e9832e52d: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:53.093368 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:53.093295    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs: secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:53.093368 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:53.093365    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials podName:bf9d8836-6fb6-459f-a05c-f95852bb9e73 nodeName:}" failed. No retries permitted until 2026-10-30 12:18:54.093346242 +0000 UTC m=+2933.313631150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs" (UID: "bf9d8836-6fb6-459f-a05c-f95852bb9e73") : secret "agc-cred-0c9e9832e52d" not found
Oct 30 12:18:53.632715 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:53.632680    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"
Oct 30 12:18:53.700202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:53.700161    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"]
Oct 30 12:18:53.705000 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:53.704969    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-6885cwpgs"]
Oct 30 12:18:53.802930 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:53.802898    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf9d8836-6fb6-459f-a05c-f95852bb9e73-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:54.689372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.689325    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:18:54.694718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.694698    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.722502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.722470    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:18:54.815780 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.815744    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxq9v\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.815940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.815923    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.816016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.815958    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.816016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.815987    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.916092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.916045    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9d8836-6fb6-459f-a05c-f95852bb9e73" path="/var/lib/kubelet/pods/bf9d8836-6fb6-459f-a05c-f95852bb9e73/volumes"
Oct 30 12:18:54.916837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.916812    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.916917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.916848    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.916917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.916873    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.917021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.917006    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxq9v\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.917612 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.917586    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.917721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.917704    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.919239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.919213    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:54.938610 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:54.938587    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxq9v\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\") pod \"payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") " pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:55.009023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.008947    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:55.318374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.318153    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:18:55.318374 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:18:55.318300    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9cf922_c74c_466f_baac_d38f0b8d7bfa.slice/crio-e3f87533012745c90f17dd7c1b8c8cb97f1fc91d9ba6058170fd555389672812 WatchSource:0}: Error finding container e3f87533012745c90f17dd7c1b8c8cb97f1fc91d9ba6058170fd555389672812: Status 404 returned error can't find the container with id e3f87533012745c90f17dd7c1b8c8cb97f1fc91d9ba6058170fd555389672812
Oct 30 12:18:55.658848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.658813    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" event={"ID":"1d9cf922-c74c-466f-baac-d38f0b8d7bfa","Type":"ContainerStarted","Data":"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"}
Oct 30 12:18:55.659007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.658867    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" event={"ID":"1d9cf922-c74c-466f-baac-d38f0b8d7bfa","Type":"ContainerStarted","Data":"e3f87533012745c90f17dd7c1b8c8cb97f1fc91d9ba6058170fd555389672812"}
Oct 30 12:18:55.659092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.659036    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:18:55.697228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.697169    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" podStartSLOduration=1.697153445 podStartE2EDuration="1.697153445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:18:55.696153433 +0000 UTC m=+2934.916438342" watchObservedRunningTime="2026-10-30 12:18:55.697153445 +0000 UTC m=+2934.917438355"
Oct 30 12:18:55.788056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:55.788023    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" podUID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" containerName="payload-processing" containerID="cri-o://fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8" gracePeriod=25
Oct 30 12:18:56.068644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.068617    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:18:56.131676 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.131634    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\") pod \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") "
Oct 30 12:18:56.131845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.131738    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\") pod \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") "
Oct 30 12:18:56.131845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.131777    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\") pod \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") "
Oct 30 12:18:56.131845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.131832    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\") pod \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\" (UID: \"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f\") "
Oct 30 12:18:56.132159 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.132135    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume" pod "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" (UID: "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:56.132742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.132161    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume" pod "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" (UID: "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:18:56.132742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.132492    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:56.132742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.132514    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:56.133974 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.133947    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8" pod "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" (UID: "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f"). InnerVolumeSpecName "kube-api-access-ws8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:56.134113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.134094    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials" pod "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" (UID: "bbbcb89c-5c6f-4769-bee8-1265fbf7e19f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:18:56.233312 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.233234    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8p8\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-kube-api-access-ws8p8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:56.233312 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.233261    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:18:56.666667 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.666633    4453 generic.go:356] "Generic (PLEG): container finished" podID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" containerID="fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8" exitCode=0
Oct 30 12:18:56.666837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.666686    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"
Oct 30 12:18:56.666837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.666716    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" event={"ID":"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f","Type":"ContainerDied","Data":"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"}
Oct 30 12:18:56.666837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.666775    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr" event={"ID":"bbbcb89c-5c6f-4769-bee8-1265fbf7e19f","Type":"ContainerDied","Data":"c1d5e24ce7cb892e70d44dce1398928a014424eabbca781352f051ac8c020da1"}
Oct 30 12:18:56.666837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.666803    4453 scope.go:122] "RemoveContainer" containerID="fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"
Oct 30 12:18:56.677720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.677704    4453 scope.go:122] "RemoveContainer" containerID="fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"
Oct 30 12:18:56.678012 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:18:56.677993    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8\": container with ID starting with fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8 not found: ID does not exist" containerID="fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"
Oct 30 12:18:56.678064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.678020    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8"} err="failed to get container status \"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8\": rpc error: code = NotFound desc = could not find container \"fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8\": container with ID starting with fd5482a0495813d8bc8777ae813abb31f5dfb0df6ec86af9980115b037ef52c8 not found: ID does not exist"
Oct 30 12:18:56.711716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.711689    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:18:56.720956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.720933    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-685bc2fdr"]
Oct 30 12:18:56.912043 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:18:56.912010    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" path="/var/lib/kubelet/pods/bbbcb89c-5c6f-4769-bee8-1265fbf7e19f/volumes"
Oct 30 12:19:00.968755 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.968722    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:19:00.969434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.969419    4453 cpu_manager.go:414] "containerMap: removing container" podUID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" containerName="payload-processing"
Oct 30 12:19:00.969478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.969435    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" containerName="payload-processing"
Oct 30 12:19:00.969550 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.969541    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbcb89c-5c6f-4769-bee8-1265fbf7e19f" containerName="payload-processing"
Oct 30 12:19:00.974775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.974750    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:00.987468 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:00.987445    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7b8a9d-openshift-default-dockercfg-rfjd2\""
Oct 30 12:19:01.004137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.004107    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:19:01.103027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.102990    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103067    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103159    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103204    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103400 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103383    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnjj\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103436    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103469    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103529 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103494    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.103592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.103580    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204459    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204605 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204510    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204605 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204589    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204632    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204731    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnjj\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204773    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204799    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204825    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204875    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.204958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204879    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.205074 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.204967    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.205143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.205108    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.205191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.205147    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.205702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.205679    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.206964 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.206948    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.207187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.207169    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.225141 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.225082    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.225256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.225206    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnjj\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\") pod \"e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") " pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.289918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.289895    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:01.604448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.604422    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:19:01.606100 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:19:01.606077    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5778e7e4_78ba_48ce_b019_ca52cd3c0aa4.slice/crio-c9050f0a4d37d64f02d07a00511f0ef75354abc2bae7230bb2414b2b384f093d WatchSource:0}: Error finding container c9050f0a4d37d64f02d07a00511f0ef75354abc2bae7230bb2414b2b384f093d: Status 404 returned error can't find the container with id c9050f0a4d37d64f02d07a00511f0ef75354abc2bae7230bb2414b2b384f093d
Oct 30 12:19:01.608889 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.608853    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:19:01.608979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.608916    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:19:01.608979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.608946    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:19:01.705166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.705136    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" event={"ID":"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4","Type":"ContainerStarted","Data":"3517314312c04041ba3cbaed1e34ef6d72adcc78c65289c517de9aaa9e6278a0"}
Oct 30 12:19:01.705320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.705187    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" event={"ID":"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4","Type":"ContainerStarted","Data":"c9050f0a4d37d64f02d07a00511f0ef75354abc2bae7230bb2414b2b384f093d"}
Oct 30 12:19:01.748111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:01.747652    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podStartSLOduration=1.747632253 podStartE2EDuration="1.747632253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:01.744980613 +0000 UTC m=+2940.965265524" watchObservedRunningTime="2026-10-30 12:19:01.747632253 +0000 UTC m=+2940.967917167"
Oct 30 12:19:02.290918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:02.290881    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:02.292741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:02.292714    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:02.292886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:02.292778    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:03.313518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:03.313476    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:03.313978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:03.313556    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:04.306364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:04.305795    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:04.306364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:04.305855    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:05.299356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:05.296663    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:05.299356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:05.296727    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:06.292677 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:06.292636    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:06.293137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:06.293097    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:06.412355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:06.400845    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:19:06.412355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:06.410001    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:19:06.421484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:06.421461    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:19:07.321357 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:07.314667    4453 patch_prober.go:29] interesting pod/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 12:19:07.321357 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:07.314730    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 12:19:07.587361 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:19:07.582037    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:19:08.757291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:08.757256    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:08.757913 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:08.757889    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:08.758680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:08.758660    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:19:11.401251 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.401201    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" podUID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" containerName="payload-processing" containerID="cri-o://427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238" gracePeriod=25
Oct 30 12:19:11.410854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.410821    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" podUID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" containerName="payload-processing" containerID="cri-o://e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d" gracePeriod=25
Oct 30 12:19:11.687940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.687909    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:19:11.706868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.706841    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:19:11.788920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.788885    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\") pod \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") "
Oct 30 12:19:11.789105 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789047    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\") pod \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") "
Oct 30 12:19:11.789170 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789114    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\") pod \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") "
Oct 30 12:19:11.789170 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789159    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\") pod \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") "
Oct 30 12:19:11.789289 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789215    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\") pod \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") "
Oct 30 12:19:11.789289 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789252    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\") pod \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") "
Oct 30 12:19:11.789431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789304    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\") pod \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\" (UID: \"1d9cf922-c74c-466f-baac-d38f0b8d7bfa\") "
Oct 30 12:19:11.789431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.789353    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\") pod \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\" (UID: \"b32ae31b-e0e6-430d-89d0-f79c899af1d2\") "
Oct 30 12:19:11.790055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.790018    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume" pod "1d9cf922-c74c-466f-baac-d38f0b8d7bfa" (UID: "1d9cf922-c74c-466f-baac-d38f0b8d7bfa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:11.790421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.790397    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume" pod "b32ae31b-e0e6-430d-89d0-f79c899af1d2" (UID: "b32ae31b-e0e6-430d-89d0-f79c899af1d2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:11.790550 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.790491    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume" pod "1d9cf922-c74c-466f-baac-d38f0b8d7bfa" (UID: "1d9cf922-c74c-466f-baac-d38f0b8d7bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:11.792724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.792697    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume" pod "b32ae31b-e0e6-430d-89d0-f79c899af1d2" (UID: "b32ae31b-e0e6-430d-89d0-f79c899af1d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:19:11.792724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.792704    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials" pod "b32ae31b-e0e6-430d-89d0-f79c899af1d2" (UID: "b32ae31b-e0e6-430d-89d0-f79c899af1d2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:11.792899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.792725    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk" pod "b32ae31b-e0e6-430d-89d0-f79c899af1d2" (UID: "b32ae31b-e0e6-430d-89d0-f79c899af1d2"). InnerVolumeSpecName "kube-api-access-x4jvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:11.793457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.793144    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials" pod "1d9cf922-c74c-466f-baac-d38f0b8d7bfa" (UID: "1d9cf922-c74c-466f-baac-d38f0b8d7bfa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:11.794373 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.794345    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v" pod "1d9cf922-c74c-466f-baac-d38f0b8d7bfa" (UID: "1d9cf922-c74c-466f-baac-d38f0b8d7bfa"). InnerVolumeSpecName "kube-api-access-mxq9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:19:11.794950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.794915    4453 generic.go:356] "Generic (PLEG): container finished" podID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" containerID="427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238" exitCode=0
Oct 30 12:19:11.795045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.794969    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"
Oct 30 12:19:11.795045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.795002    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" event={"ID":"1d9cf922-c74c-466f-baac-d38f0b8d7bfa","Type":"ContainerDied","Data":"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"}
Oct 30 12:19:11.795157 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.795084    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5" event={"ID":"1d9cf922-c74c-466f-baac-d38f0b8d7bfa","Type":"ContainerDied","Data":"e3f87533012745c90f17dd7c1b8c8cb97f1fc91d9ba6058170fd555389672812"}
Oct 30 12:19:11.795216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.795177    4453 scope.go:122] "RemoveContainer" containerID="427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"
Oct 30 12:19:11.802088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.800560    4453 generic.go:356] "Generic (PLEG): container finished" podID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" containerID="e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d" exitCode=0
Oct 30 12:19:11.802088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.800747    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" event={"ID":"b32ae31b-e0e6-430d-89d0-f79c899af1d2","Type":"ContainerDied","Data":"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"}
Oct 30 12:19:11.802088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.800791    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9" event={"ID":"b32ae31b-e0e6-430d-89d0-f79c899af1d2","Type":"ContainerDied","Data":"2572adb7550722edd0abc86d5c2306a7e7541fafb1aba54eae722a2c270ea76c"}
Oct 30 12:19:11.802088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.800865    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"
Oct 30 12:19:11.811613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.811597    4453 scope.go:122] "RemoveContainer" containerID="427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"
Oct 30 12:19:11.811879 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:19:11.811863    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238\": container with ID starting with 427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238 not found: ID does not exist" containerID="427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"
Oct 30 12:19:11.811929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.811886    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238"} err="failed to get container status \"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238\": rpc error: code = NotFound desc = could not find container \"427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238\": container with ID starting with 427e2c52b4978578859221e0b550b18caf35a31a779969e2d4cb8fba67ef9238 not found: ID does not exist"
Oct 30 12:19:11.811929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.811902    4453 scope.go:122] "RemoveContainer" containerID="e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"
Oct 30 12:19:11.820606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.820590    4453 scope.go:122] "RemoveContainer" containerID="e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"
Oct 30 12:19:11.820856 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:19:11.820836    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d\": container with ID starting with e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d not found: ID does not exist" containerID="e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"
Oct 30 12:19:11.820914 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.820864    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d"} err="failed to get container status \"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d\": rpc error: code = NotFound desc = could not find container \"e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d\": container with ID starting with e5017a4c1a38408347196d2d6487557275cadac05e13ed19a9e20680634b727d not found: ID does not exist"
Oct 30 12:19:11.871415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.871381    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:19:11.875345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.875301    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-59d7g2pl5"]
Oct 30 12:19:11.891439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891404    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jvk\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-kube-api-access-x4jvk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891438    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891454    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891466    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891479    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32ae31b-e0e6-430d-89d0-f79c899af1d2-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891492    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxq9v\" (UniqueName: \"kubernetes.io/projected/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-kube-api-access-mxq9v\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891504    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d9cf922-c74c-466f-baac-d38f0b8d7bfa-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.891637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.891521    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32ae31b-e0e6-430d-89d0-f79c899af1d2-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:19:11.894826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.894799    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:19:11.898656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:11.898634    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b8a9d/payload-processing-external-model-e2e-extmod-b-7b8a9d-8569s4kn9"]
Oct 30 12:19:12.911879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:12.911843    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" path="/var/lib/kubelet/pods/1d9cf922-c74c-466f-baac-d38f0b8d7bfa/volumes"
Oct 30 12:19:12.912288 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:12.912272    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" path="/var/lib/kubelet/pods/b32ae31b-e0e6-430d-89d0-f79c899af1d2/volumes"
Oct 30 12:19:15.739235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.739198    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:19:15.740136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740109    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" containerName="payload-processing"
Oct 30 12:19:15.740258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740137    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" containerName="payload-processing"
Oct 30 12:19:15.740258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740172    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" containerName="payload-processing"
Oct 30 12:19:15.740258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740180    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" containerName="payload-processing"
Oct 30 12:19:15.740489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740349    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9cf922-c74c-466f-baac-d38f0b8d7bfa" containerName="payload-processing"
Oct 30 12:19:15.740489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.740371    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32ae31b-e0e6-430d-89d0-f79c899af1d2" containerName="payload-processing"
Oct 30 12:19:15.745641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.745620    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.760542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.760521    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5c638a89dd61\""
Oct 30 12:19:15.763118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.763099    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d6f15183adb0\""
Oct 30 12:19:15.770361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.770323    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:19:15.770456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.770390    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qtlnm\""
Oct 30 12:19:15.770456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.770413    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:19:15.772633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.772610    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:19:15.772633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.772626    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:19:15.776047 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.776023    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:19:15.841621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.841591    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.841765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.841632    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.841765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.841660    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.841765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.841742    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfw94\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.942432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.942400    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.942587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.942435    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.942587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.942455    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.942587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.942489    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfw94\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.943180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.943157    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.943278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.943176    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.944989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.944968    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:15.963088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:15.963062    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfw94\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\") pod \"payload-processing-external-model-975fcdbd8-xblrk\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") " pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:16.060227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.060159    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:16.366068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.366038    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:19:16.367832 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:19:16.367796    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7e1507_2c90_4976_9bdd_1da54db7200d.slice/crio-25c135006ab4178f030051be4ddd21b581fef05d1ee629b0476fc5eda83b0eef WatchSource:0}: Error finding container 25c135006ab4178f030051be4ddd21b581fef05d1ee629b0476fc5eda83b0eef: Status 404 returned error can't find the container with id 25c135006ab4178f030051be4ddd21b581fef05d1ee629b0476fc5eda83b0eef
Oct 30 12:19:16.838920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.838818    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" event={"ID":"ce7e1507-2c90-4976-9bdd-1da54db7200d","Type":"ContainerStarted","Data":"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"}
Oct 30 12:19:16.838920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.838887    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:16.838920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.838910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" event={"ID":"ce7e1507-2c90-4976-9bdd-1da54db7200d","Type":"ContainerStarted","Data":"25c135006ab4178f030051be4ddd21b581fef05d1ee629b0476fc5eda83b0eef"}
Oct 30 12:19:16.875785 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:16.875734    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" podStartSLOduration=1.8757205959999999 podStartE2EDuration="1.875720596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:16.874038861 +0000 UTC m=+2956.094323771" watchObservedRunningTime="2026-10-30 12:19:16.875720596 +0000 UTC m=+2956.096005506"
Oct 30 12:19:27.848638 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:27.848608    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:19:37.571016 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:19:37.570974    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:19:55.383934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:55.383898    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:19:57.107740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.107703    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:19:57.113371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.113346    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.138222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.138200    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cwb2f\""
Oct 30 12:19:57.138363 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.138258    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a33ca779fec9\""
Oct 30 12:19:57.148023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.147996    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:19:57.191315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.191278    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glp2p\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.191315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.191317    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.191586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.191467    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.191740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.191714    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.293260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.293220    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glp2p\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.293568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.293548    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.293733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.293718    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.294065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.294050    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.295217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.295173    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.296254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.296210    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.314743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.314680    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.336951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.336907    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glp2p\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\") pod \"payload-processing-external-model-c985fc668-s84bg\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") " pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.434668 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.434560    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:57.759089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:57.759059    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:19:58.144147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:58.144109    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" event={"ID":"119d8d60-35ac-44b0-8cf3-4c3c253e29ce","Type":"ContainerStarted","Data":"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"}
Oct 30 12:19:58.144586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:58.144179    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:19:58.144586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:58.144205    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" event={"ID":"119d8d60-35ac-44b0-8cf3-4c3c253e29ce","Type":"ContainerStarted","Data":"b5a83fefe4e68f8f40b17640534b0996f595d039d82b592a332696f02a66e5c2"}
Oct 30 12:19:58.194354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:19:58.193979    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" podStartSLOduration=1.193962255 podStartE2EDuration="1.193962255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:19:58.190239708 +0000 UTC m=+2997.410524620" watchObservedRunningTime="2026-10-30 12:19:58.193962255 +0000 UTC m=+2997.414247166"
Oct 30 12:20:00.384478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.384441    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" podUID="ce7e1507-2c90-4976-9bdd-1da54db7200d" containerName="payload-processing" containerID="cri-o://1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647" gracePeriod=25
Oct 30 12:20:00.705288 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.705265    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:20:00.747986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.747958    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\") pod \"ce7e1507-2c90-4976-9bdd-1da54db7200d\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") "
Oct 30 12:20:00.749013 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.748491    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume" pod "ce7e1507-2c90-4976-9bdd-1da54db7200d" (UID: "ce7e1507-2c90-4976-9bdd-1da54db7200d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:00.749168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.748988    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\") pod \"ce7e1507-2c90-4976-9bdd-1da54db7200d\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") "
Oct 30 12:20:00.749576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.749235    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\") pod \"ce7e1507-2c90-4976-9bdd-1da54db7200d\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") "
Oct 30 12:20:00.749576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.749318    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\") pod \"ce7e1507-2c90-4976-9bdd-1da54db7200d\" (UID: \"ce7e1507-2c90-4976-9bdd-1da54db7200d\") "
Oct 30 12:20:00.749576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.749460    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume" pod "ce7e1507-2c90-4976-9bdd-1da54db7200d" (UID: "ce7e1507-2c90-4976-9bdd-1da54db7200d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:00.750750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.750323    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:00.750750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.750374    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce7e1507-2c90-4976-9bdd-1da54db7200d-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:00.759369 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.758315    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials" pod "ce7e1507-2c90-4976-9bdd-1da54db7200d" (UID: "ce7e1507-2c90-4976-9bdd-1da54db7200d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:00.759369 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.758534    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94" pod "ce7e1507-2c90-4976-9bdd-1da54db7200d" (UID: "ce7e1507-2c90-4976-9bdd-1da54db7200d"). InnerVolumeSpecName "kube-api-access-mfw94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:00.773746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.773720    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:20:00.774740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.774721    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ce7e1507-2c90-4976-9bdd-1da54db7200d" containerName="payload-processing"
Oct 30 12:20:00.774805 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.774743    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7e1507-2c90-4976-9bdd-1da54db7200d" containerName="payload-processing"
Oct 30 12:20:00.774972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.774959    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7e1507-2c90-4976-9bdd-1da54db7200d" containerName="payload-processing"
Oct 30 12:20:00.779345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.779302    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.797166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797138    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"agc-cred-22c2517855b2\""
Oct 30 12:20:00.797280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797201    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"routing-overlay\""
Oct 30 12:20:00.797280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797277    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"kube-root-ca.crt\""
Oct 30 12:20:00.797449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797312    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-e2e--fe4784b3-dockercfg-mq9mq\""
Oct 30 12:20:00.797449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797320    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-59b8a9\""
Oct 30 12:20:00.797449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797369    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"openshift-service-ca.crt\""
Oct 30 12:20:00.797858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.797837    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:20:00.851770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.851741    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8cd\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.851940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.851821    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.851940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.851928    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.852063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.851951    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.852063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.852054    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfw94\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-kube-api-access-mfw94\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:00.852166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.852065    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce7e1507-2c90-4976-9bdd-1da54db7200d-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:00.966590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.966378    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.966766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.966641    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.966766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.966685    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.969518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.969485    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8cd\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.983064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.983005    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"routing-overlay\""
Oct 30 12:20:00.983194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.983122    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"agc-cred-22c2517855b2\""
Oct 30 12:20:00.983256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.983243    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-59b8a9\""
Oct 30 12:20:00.988262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.988226    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.991634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.991602    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:00.993543 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:00.993522    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:01.000878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.000859    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"kube-root-ca.crt\""
Oct 30 12:20:01.021132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.021113    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"openshift-service-ca.crt\""
Oct 30 12:20:01.032772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.032751    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8cd\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:01.110122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.110092    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"payload-processing-external-model-e2e--fe4784b3-dockercfg-mq9mq\""
Oct 30 12:20:01.119360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.119326    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:01.167476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.167444    4453 generic.go:356] "Generic (PLEG): container finished" podID="ce7e1507-2c90-4976-9bdd-1da54db7200d" containerID="1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647" exitCode=0
Oct 30 12:20:01.167626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.167519    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"
Oct 30 12:20:01.167626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.167537    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" event={"ID":"ce7e1507-2c90-4976-9bdd-1da54db7200d","Type":"ContainerDied","Data":"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"}
Oct 30 12:20:01.167626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.167611    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk" event={"ID":"ce7e1507-2c90-4976-9bdd-1da54db7200d","Type":"ContainerDied","Data":"25c135006ab4178f030051be4ddd21b581fef05d1ee629b0476fc5eda83b0eef"}
Oct 30 12:20:01.167796 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.167648    4453 scope.go:122] "RemoveContainer" containerID="1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"
Oct 30 12:20:01.197501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.194920    4453 scope.go:122] "RemoveContainer" containerID="1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"
Oct 30 12:20:01.197586 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:01.197487    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647\": container with ID starting with 1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647 not found: ID does not exist" containerID="1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"
Oct 30 12:20:01.197586 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.197540    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647"} err="failed to get container status \"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647\": rpc error: code = NotFound desc = could not find container \"1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647\": container with ID starting with 1233ae2d59732f0f0fc2dc14b2d317f3ed981588813bcc5899b428b2da206647 not found: ID does not exist"
Oct 30 12:20:01.217486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.217429    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:20:01.221864 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.221839    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-975fcdbd8-xblrk"]
Oct 30 12:20:01.427268 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.427236    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:20:01.428175 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:01.428150    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c69550_a1f7_4ec8_a892_ba95db52e8d6.slice/crio-c05bf4b1be4d9e0745a976728c3f185e83df5a623db6214c679e7d2f2cd54c57 WatchSource:0}: Error finding container c05bf4b1be4d9e0745a976728c3f185e83df5a623db6214c679e7d2f2cd54c57: Status 404 returned error can't find the container with id c05bf4b1be4d9e0745a976728c3f185e83df5a623db6214c679e7d2f2cd54c57
Oct 30 12:20:01.676842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.676803    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:20:01.681298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.681278    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.698423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698393    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"openshift-service-ca.crt\""
Oct 30 12:20:01.698558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698490    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b15ec3\""
Oct 30 12:20:01.698783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698763    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"kube-root-ca.crt\""
Oct 30 12:20:01.698942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698926    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"payload-processing-external-model-e2e--a1d927ff-dockercfg-v64td\""
Oct 30 12:20:01.698983 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698931    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"agc-cred-3bed90f862d4\""
Oct 30 12:20:01.698983 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.698970    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"routing-overlay\""
Oct 30 12:20:01.702116 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.702087    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:20:01.892614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.892584    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fv2\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.892752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.892706    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.892752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.892735    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.892973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.892949    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.994431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.994399    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.994589 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.994482    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fv2\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.994589 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.994555    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.994589 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.994575    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.995192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.995165    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.995192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.995186    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:01.997149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:01.997125    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:02.015283 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.015256    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fv2\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:02.188946 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.188906    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" event={"ID":"72c69550-a1f7-4ec8-a892-ba95db52e8d6","Type":"ContainerStarted","Data":"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"}
Oct 30 12:20:02.189133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.189003    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" event={"ID":"72c69550-a1f7-4ec8-a892-ba95db52e8d6","Type":"ContainerStarted","Data":"c05bf4b1be4d9e0745a976728c3f185e83df5a623db6214c679e7d2f2cd54c57"}
Oct 30 12:20:02.189184 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.189140    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:02.229087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.229028    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" podStartSLOduration=2.229010321 podStartE2EDuration="2.229010321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:02.225608667 +0000 UTC m=+3001.445893573" watchObservedRunningTime="2026-10-30 12:20:02.229010321 +0000 UTC m=+3001.449295235"
Oct 30 12:20:02.295197 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.295154    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:02.600722 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.600685    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:20:02.601758 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:02.601730    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e5dd90_5e58_48ab_8ef1_1a357e8a5536.slice/crio-57d6eb1c56835c0adcc9c2c69c8ba76ee0730d6ca579b95253991e6d7aa478be WatchSource:0}: Error finding container 57d6eb1c56835c0adcc9c2c69c8ba76ee0730d6ca579b95253991e6d7aa478be: Status 404 returned error can't find the container with id 57d6eb1c56835c0adcc9c2c69c8ba76ee0730d6ca579b95253991e6d7aa478be
Oct 30 12:20:02.710733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.710687    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:20:02.716464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.716425    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:20:02.913502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:02.913468    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7e1507-2c90-4976-9bdd-1da54db7200d" path="/var/lib/kubelet/pods/ce7e1507-2c90-4976-9bdd-1da54db7200d/volumes"
Oct 30 12:20:03.197806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:03.197712    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" event={"ID":"31e5dd90-5e58-48ab-8ef1-1a357e8a5536","Type":"ContainerStarted","Data":"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"}
Oct 30 12:20:03.197806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:03.197767    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" event={"ID":"31e5dd90-5e58-48ab-8ef1-1a357e8a5536","Type":"ContainerStarted","Data":"57d6eb1c56835c0adcc9c2c69c8ba76ee0730d6ca579b95253991e6d7aa478be"}
Oct 30 12:20:03.198051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:03.198019    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:03.234795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:03.234749    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" podStartSLOduration=2.234736507 podStartE2EDuration="2.234736507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:03.233648563 +0000 UTC m=+3002.453933469" watchObservedRunningTime="2026-10-30 12:20:03.234736507 +0000 UTC m=+3002.455021417"
Oct 30 12:20:05.780350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.780303    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:05.785623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.785600    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.811009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.810967    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4f7443\""
Oct 30 12:20:05.811009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.810995    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"openshift-service-ca.crt\""
Oct 30 12:20:05.811411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.811014    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"kube-root-ca.crt\""
Oct 30 12:20:05.811509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.811409    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"payload-processing-external-model-e2e--9f9529d9-dockercfg-fdxmq\""
Oct 30 12:20:05.811509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.811258    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-010fd4c5b4d7\""
Oct 30 12:20:05.811509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.811372    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"agc-cred-097f9a689008\""
Oct 30 12:20:05.811509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.811436    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4f7443\"/\"routing-overlay\""
Oct 30 12:20:05.818598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.818569    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:05.851719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.851681    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.851948 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.851913    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.852037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.851965    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.852096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.852063    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdpr\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.953037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953004    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.953254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953074    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.953254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953192    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdpr\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.953396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953374    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.953893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953864    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.954008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.953945    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.955787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.955766    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:05.974183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:05.974158    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdpr\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:06.099903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:06.099817    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:06.405927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:06.405896    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:06.407809 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:06.407777    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396dfda7_0fbd_415d_ba8a_187caa617eb6.slice/crio-d5019ddd5a9317b2879635e51d79dea2b1fa47864d4b013a62e903466cdcc05f WatchSource:0}: Error finding container d5019ddd5a9317b2879635e51d79dea2b1fa47864d4b013a62e903466cdcc05f: Status 404 returned error can't find the container with id d5019ddd5a9317b2879635e51d79dea2b1fa47864d4b013a62e903466cdcc05f
Oct 30 12:20:07.258019 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.257978    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" event={"ID":"396dfda7-0fbd-415d-ba8a-187caa617eb6","Type":"ContainerStarted","Data":"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"}
Oct 30 12:20:07.258596 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.258570    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" event={"ID":"396dfda7-0fbd-415d-ba8a-187caa617eb6","Type":"ContainerStarted","Data":"d5019ddd5a9317b2879635e51d79dea2b1fa47864d4b013a62e903466cdcc05f"}
Oct 30 12:20:07.259733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.259712    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:07.303492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.303439    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" podStartSLOduration=2.303425058 podStartE2EDuration="2.303425058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:07.29743562 +0000 UTC m=+3006.517720533" watchObservedRunningTime="2026-10-30 12:20:07.303425058 +0000 UTC m=+3006.523709967"
Oct 30 12:20:07.376856 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.376819    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4f7443/agc-cred-010fd4c5b4d7: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.376856 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.376857    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.377065 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.376921    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials podName:396dfda7-0fbd-415d-ba8a-187caa617eb6 nodeName:}" failed. No retries permitted until 2026-10-30 12:20:07.876899392 +0000 UTC m=+3007.097184283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6") : secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.572709 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.572624    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:20:07.711711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.711673    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" podUID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" containerName="payload-processing" containerID="cri-o://52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d" gracePeriod=25
Oct 30 12:20:07.884623 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.884592    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4f7443/agc-cred-010fd4c5b4d7: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.884737 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.884631    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.884737 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:07.884692    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials podName:396dfda7-0fbd-415d-ba8a-187caa617eb6 nodeName:}" failed. No retries permitted until 2026-10-30 12:20:08.884674852 +0000 UTC m=+3008.104959740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6") : secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:07.981806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:07.981777    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:20:08.087591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.087521    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\") pod \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") "
Oct 30 12:20:08.087591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.087555    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\") pod \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") "
Oct 30 12:20:08.087795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.087776    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\") pod \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") "
Oct 30 12:20:08.087870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.087855    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\") pod \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\" (UID: \"119d8d60-35ac-44b0-8cf3-4c3c253e29ce\") "
Oct 30 12:20:08.087971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.087943    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume" pod "119d8d60-35ac-44b0-8cf3-4c3c253e29ce" (UID: "119d8d60-35ac-44b0-8cf3-4c3c253e29ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:08.088035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.088023    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume" pod "119d8d60-35ac-44b0-8cf3-4c3c253e29ce" (UID: "119d8d60-35ac-44b0-8cf3-4c3c253e29ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:08.088403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.088381    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:08.088403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.088399    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:08.089989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.089961    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p" pod "119d8d60-35ac-44b0-8cf3-4c3c253e29ce" (UID: "119d8d60-35ac-44b0-8cf3-4c3c253e29ce"). InnerVolumeSpecName "kube-api-access-glp2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:08.090117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.090087    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials" pod "119d8d60-35ac-44b0-8cf3-4c3c253e29ce" (UID: "119d8d60-35ac-44b0-8cf3-4c3c253e29ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:08.190132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.190101    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glp2p\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-kube-api-access-glp2p\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:08.190132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.190129    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/119d8d60-35ac-44b0-8cf3-4c3c253e29ce-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:08.265204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.265167    4453 generic.go:356] "Generic (PLEG): container finished" podID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" containerID="52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d" exitCode=0
Oct 30 12:20:08.265711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.265224    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg"
Oct 30 12:20:08.265711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.265249    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" event={"ID":"119d8d60-35ac-44b0-8cf3-4c3c253e29ce","Type":"ContainerDied","Data":"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"}
Oct 30 12:20:08.265711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.265316    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c985fc668-s84bg" event={"ID":"119d8d60-35ac-44b0-8cf3-4c3c253e29ce","Type":"ContainerDied","Data":"b5a83fefe4e68f8f40b17640534b0996f595d039d82b592a332696f02a66e5c2"}
Oct 30 12:20:08.265711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.265374    4453 scope.go:122] "RemoveContainer" containerID="52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"
Oct 30 12:20:08.280795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.280689    4453 scope.go:122] "RemoveContainer" containerID="52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"
Oct 30 12:20:08.282131 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:08.281346    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d\": container with ID starting with 52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d not found: ID does not exist" containerID="52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"
Oct 30 12:20:08.282131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.281399    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d"} err="failed to get container status \"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d\": rpc error: code = NotFound desc = could not find container \"52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d\": container with ID starting with 52052c7811a7a5a8c56918f924fbebec45a057ebfdfa1ef4cc618d5c78f3026d not found: ID does not exist"
Oct 30 12:20:08.345000 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.344911    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:20:08.363620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.363590    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c985fc668-s84bg"]
Oct 30 12:20:08.913026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:08.912988    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" path="/var/lib/kubelet/pods/119d8d60-35ac-44b0-8cf3-4c3c253e29ce/volumes"
Oct 30 12:20:08.919390 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:08.919367    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4f7443/agc-cred-010fd4c5b4d7: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:08.919480 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:08.919396    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:08.919480 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:08.919453    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials podName:396dfda7-0fbd-415d-ba8a-187caa617eb6 nodeName:}" failed. No retries permitted until 2026-10-30 12:20:10.919436438 +0000 UTC m=+3010.139721331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6") : secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:09.180147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.180063    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:09.214343 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.214306    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:09.215006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.214991    4453 cpu_manager.go:414] "containerMap: removing container" podUID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" containerName="payload-processing"
Oct 30 12:20:09.215069 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.215007    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" containerName="payload-processing"
Oct 30 12:20:09.215149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.215139    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="119d8d60-35ac-44b0-8cf3-4c3c253e29ce" containerName="payload-processing"
Oct 30 12:20:09.220023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.220006    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.239341 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.239312    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:09.324127 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.324095    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.324553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.324132    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.324553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.324223    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xtq\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.324553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.324474    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.425721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.425687    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xtq\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.425912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.425892    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.425995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.425954    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.425995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.425989    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.426769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.426740    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.427224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.427199    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.428648 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.428626    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.446231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.446170    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xtq\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\") pod \"payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") " pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.534671 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.534641    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:09.846246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:09.846219    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:09.847508 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:09.847476    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5eac023_f89d_4c9b_bcdd_f75c0bf10f33.slice/crio-db5ed0e62496deb28d9a8130407782ec0ff5bfa9c6cf96a069fa2800957c55db WatchSource:0}: Error finding container db5ed0e62496deb28d9a8130407782ec0ff5bfa9c6cf96a069fa2800957c55db: Status 404 returned error can't find the container with id db5ed0e62496deb28d9a8130407782ec0ff5bfa9c6cf96a069fa2800957c55db
Oct 30 12:20:10.283064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:10.283031    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" event={"ID":"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33","Type":"ContainerStarted","Data":"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"}
Oct 30 12:20:10.283307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:10.283088    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" event={"ID":"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33","Type":"ContainerStarted","Data":"db5ed0e62496deb28d9a8130407782ec0ff5bfa9c6cf96a069fa2800957c55db"}
Oct 30 12:20:10.283307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:10.283134    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:10.361343 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:10.361214    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" podStartSLOduration=1.361194451 podStartE2EDuration="1.361194451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:10.359524988 +0000 UTC m=+3009.579809899" watchObservedRunningTime="2026-10-30 12:20:10.361194451 +0000 UTC m=+3009.581479364"
Oct 30 12:20:10.952428 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:10.952400    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4f7443/agc-cred-010fd4c5b4d7: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:10.952428 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:10.952428    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:10.952644 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:10.952480    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials podName:396dfda7-0fbd-415d-ba8a-187caa617eb6 nodeName:}" failed. No retries permitted until 2026-10-30 12:20:14.952465801 +0000 UTC m=+3014.172750690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6") : secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:13.201154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:13.201125    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:20:14.207788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:14.207714    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:20:15.029701 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:15.029661    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4f7443/agc-cred-010fd4c5b4d7: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:15.029701 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:15.029692    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d: secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:15.029911 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:15.029742    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials podName:396dfda7-0fbd-415d-ba8a-187caa617eb6 nodeName:}" failed. No retries permitted until 2026-10-30 12:20:23.029727423 +0000 UTC m=+3022.250012324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6") : secret "agc-cred-010fd4c5b4d7" not found
Oct 30 12:20:15.284082 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.283966    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" podUID="396dfda7-0fbd-415d-ba8a-187caa617eb6" containerName="payload-processing" containerID="cri-o://ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263" gracePeriod=25
Oct 30 12:20:15.547025 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.546966    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:15.637224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637191    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\") pod \"396dfda7-0fbd-415d-ba8a-187caa617eb6\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") "
Oct 30 12:20:15.637439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637276    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\") pod \"396dfda7-0fbd-415d-ba8a-187caa617eb6\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") "
Oct 30 12:20:15.637439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637398    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\") pod \"396dfda7-0fbd-415d-ba8a-187caa617eb6\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") "
Oct 30 12:20:15.637675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637480    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\") pod \"396dfda7-0fbd-415d-ba8a-187caa617eb6\" (UID: \"396dfda7-0fbd-415d-ba8a-187caa617eb6\") "
Oct 30 12:20:15.638006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637886    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume" pod "396dfda7-0fbd-415d-ba8a-187caa617eb6" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:15.638006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.637947    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume" pod "396dfda7-0fbd-415d-ba8a-187caa617eb6" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:15.638146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.638048    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:15.639618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.639590    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr" pod "396dfda7-0fbd-415d-ba8a-187caa617eb6" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6"). InnerVolumeSpecName "kube-api-access-cvdpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:15.639706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.639626    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials" pod "396dfda7-0fbd-415d-ba8a-187caa617eb6" (UID: "396dfda7-0fbd-415d-ba8a-187caa617eb6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:15.739592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.739557    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/396dfda7-0fbd-415d-ba8a-187caa617eb6-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:15.739592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.739586    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdpr\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-kube-api-access-cvdpr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:15.739592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:15.739595    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/396dfda7-0fbd-415d-ba8a-187caa617eb6-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:16.288442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.288409    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:20:16.289090 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.289073    4453 cpu_manager.go:414] "containerMap: removing container" podUID="396dfda7-0fbd-415d-ba8a-187caa617eb6" containerName="payload-processing"
Oct 30 12:20:16.289147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.289091    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396dfda7-0fbd-415d-ba8a-187caa617eb6" containerName="payload-processing"
Oct 30 12:20:16.289224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.289214    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="396dfda7-0fbd-415d-ba8a-187caa617eb6" containerName="payload-processing"
Oct 30 12:20:16.293037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.293019    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.305309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.305286    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-5f28e2f75bbb\""
Oct 30 12:20:16.305460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.305379    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-5d74e357e0f1\""
Oct 30 12:20:16.315082 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.315049    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"routing-overlay\""
Oct 30 12:20:16.315194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.315085    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"kube-root-ca.crt\""
Oct 30 12:20:16.315194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.315128    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-e2e--8475db1b-dockercfg-xhptp\""
Oct 30 12:20:16.315311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.315227    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"openshift-service-ca.crt\""
Oct 30 12:20:16.315487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.315469    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f8f132\""
Oct 30 12:20:16.319542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.319519    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:20:16.329738 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.329711    4453 generic.go:356] "Generic (PLEG): container finished" podID="396dfda7-0fbd-415d-ba8a-187caa617eb6" containerID="ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263" exitCode=0
Oct 30 12:20:16.329860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.329779    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"
Oct 30 12:20:16.329860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.329779    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" event={"ID":"396dfda7-0fbd-415d-ba8a-187caa617eb6","Type":"ContainerDied","Data":"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"}
Oct 30 12:20:16.329860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.329849    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d" event={"ID":"396dfda7-0fbd-415d-ba8a-187caa617eb6","Type":"ContainerDied","Data":"d5019ddd5a9317b2879635e51d79dea2b1fa47864d4b013a62e903466cdcc05f"}
Oct 30 12:20:16.330038 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.329881    4453 scope.go:122] "RemoveContainer" containerID="ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"
Oct 30 12:20:16.343222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.343200    4453 scope.go:122] "RemoveContainer" containerID="ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"
Oct 30 12:20:16.343524 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:16.343504    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263\": container with ID starting with ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263 not found: ID does not exist" containerID="ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"
Oct 30 12:20:16.343624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.343530    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263"} err="failed to get container status \"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263\": rpc error: code = NotFound desc = could not find container \"ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263\": container with ID starting with ed1f8d6e78ebae58d95a74bc26a01c4bf52a6ee812aa9a221d53f61f4718f263 not found: ID does not exist"
Oct 30 12:20:16.347446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.347419    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.347621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.347549    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2qnr\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.347719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.347697    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.347779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.347766    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.373926 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.373903    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:16.377731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.377712    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-7bf68hj2d"]
Oct 30 12:20:16.449582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.449551    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.449716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.449621    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2qnr\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.449775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.449763    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.449820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.449806    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.450625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.450601    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.450794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.450774    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.452053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.452029    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.470287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.470261    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2qnr\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.607972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.607948    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:16.631965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.631929    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"]
Oct 30 12:20:16.637705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.637686    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652168    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652280    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652370    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652567 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652543    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj9v\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-kube-api-access-dkj9v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652679 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652632    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652679 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652668    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652717    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652818    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.652899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.652884    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.660187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.658860    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"]
Oct 30 12:20:16.753766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753725    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj9v\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-kube-api-access-dkj9v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.753967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753790    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.753967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753832    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.753967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753878    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.753967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753909    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.754178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.753968    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.754178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.754045    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.754178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.754092    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.754178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.754143    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.754891 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.754789    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.755551 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.755527    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.755770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.755740    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.755867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.755793    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.756037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.756013    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.758475 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.758451    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.762842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.762809    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.776248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.776228    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.776840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.776819    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj9v\" (UniqueName: \"kubernetes.io/projected/c5ef9f10-6825-4798-9579-2d1f3fed2dba-kube-api-access-dkj9v\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-hpp9t\" (UID: \"c5ef9f10-6825-4798-9579-2d1f3fed2dba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:16.913125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.913051    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396dfda7-0fbd-415d-ba8a-187caa617eb6" path="/var/lib/kubelet/pods/396dfda7-0fbd-415d-ba8a-187caa617eb6/volumes"
Oct 30 12:20:16.926848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.926825    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:20:16.927898 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:16.927866    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688e42cb_19e3_4835_97b7_43e7eedcdac0.slice/crio-21583bd3a37fecc49f8a6977d311c0f0137a112b0e7292be88e979a0d7906a17 WatchSource:0}: Error finding container 21583bd3a37fecc49f8a6977d311c0f0137a112b0e7292be88e979a0d7906a17: Status 404 returned error can't find the container with id 21583bd3a37fecc49f8a6977d311c0f0137a112b0e7292be88e979a0d7906a17
Oct 30 12:20:16.968099 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:16.968078    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:17.291212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.291180    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"]
Oct 30 12:20:17.291708 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:17.291684    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ef9f10_6825_4798_9579_2d1f3fed2dba.slice/crio-2cc175454b790e2e09465700bcad8f3f8fc5eff4286ec52f74b85409b820f315 WatchSource:0}: Error finding container 2cc175454b790e2e09465700bcad8f3f8fc5eff4286ec52f74b85409b820f315: Status 404 returned error can't find the container with id 2cc175454b790e2e09465700bcad8f3f8fc5eff4286ec52f74b85409b820f315
Oct 30 12:20:17.294475 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.294439    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:17.294552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.294506    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:17.294552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.294535    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:17.338143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.338115    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" event={"ID":"c5ef9f10-6825-4798-9579-2d1f3fed2dba","Type":"ContainerStarted","Data":"2cc175454b790e2e09465700bcad8f3f8fc5eff4286ec52f74b85409b820f315"}
Oct 30 12:20:17.340934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.340911    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" event={"ID":"688e42cb-19e3-4835-97b7-43e7eedcdac0","Type":"ContainerStarted","Data":"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"}
Oct 30 12:20:17.341035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.340955    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" event={"ID":"688e42cb-19e3-4835-97b7-43e7eedcdac0","Type":"ContainerStarted","Data":"21583bd3a37fecc49f8a6977d311c0f0137a112b0e7292be88e979a0d7906a17"}
Oct 30 12:20:17.341035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.340982    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:17.379714 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:17.379663    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" podStartSLOduration=1.379649623 podStartE2EDuration="1.379649623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:17.376594521 +0000 UTC m=+3016.596879431" watchObservedRunningTime="2026-10-30 12:20:17.379649623 +0000 UTC m=+3016.599934532"
Oct 30 12:20:18.348989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:18.348951    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" event={"ID":"c5ef9f10-6825-4798-9579-2d1f3fed2dba","Type":"ContainerStarted","Data":"415c8655a1eec8b54ef56cc4d4dfe2c97384fa2da660ce6be2a03414d4397a2f"}
Oct 30 12:20:18.418480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:18.418424    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podStartSLOduration=2.418403483 podStartE2EDuration="2.418403483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:18.405830819 +0000 UTC m=+3017.626115742" watchObservedRunningTime="2026-10-30 12:20:18.418403483 +0000 UTC m=+3017.638688389"
Oct 30 12:20:18.968984 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:18.968931    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:18.971009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:18.970979    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:18.971144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:18.971036    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:19.968650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:19.968617    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:19.969085 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:19.968668    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:20.370124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:20.370087    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:20.373312 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:20.373289    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:20.968503 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:20.968470    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:20.968647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:20.968523    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:21.968525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:21.968483    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:21.968991 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:21.968552    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:22.969072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:22.969028    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:22.969570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:22.969091    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:23.969439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:23.969402    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:23.969843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:23.969480    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:24.969536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:24.969496    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 12:20:24.969982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:24.969570    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 12:20:25.370515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.370468    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" podUID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" containerName="payload-processing" containerID="cri-o://027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff" gracePeriod=25
Oct 30 12:20:25.683980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.683947    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:25.821416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821359    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\") pod \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") "
Oct 30 12:20:25.821617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821497    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\") pod \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") "
Oct 30 12:20:25.821617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821595    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\") pod \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") "
Oct 30 12:20:25.821948 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821919    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\") pod \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\" (UID: \"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33\") "
Oct 30 12:20:25.822102 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821948    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume" pod "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" (UID: "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:25.822102 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.821977    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume" pod "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" (UID: "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:20:25.822507 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.822481    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:25.822613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.822511    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:25.824053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.824026    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials" pod "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" (UID: "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:25.824266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.824238    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq" pod "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" (UID: "f5eac023-f89d-4c9b-bcdd-f75c0bf10f33"). InnerVolumeSpecName "kube-api-access-b2xtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:20:25.924725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.924627    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:25.924725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:25.924670    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xtq\" (UniqueName: \"kubernetes.io/projected/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33-kube-api-access-b2xtq\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:20:26.487350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.485382    4453 generic.go:356] "Generic (PLEG): container finished" podID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" containerID="027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff" exitCode=0
Oct 30 12:20:26.487350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.485561    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" event={"ID":"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33","Type":"ContainerDied","Data":"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"}
Oct 30 12:20:26.487350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.485612    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg" event={"ID":"f5eac023-f89d-4c9b-bcdd-f75c0bf10f33","Type":"ContainerDied","Data":"db5ed0e62496deb28d9a8130407782ec0ff5bfa9c6cf96a069fa2800957c55db"}
Oct 30 12:20:26.487350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.485642    4453 scope.go:122] "RemoveContainer" containerID="027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"
Oct 30 12:20:26.487350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.485816    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"
Oct 30 12:20:26.571359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.570797    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:26.578358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.576923    4453 scope.go:122] "RemoveContainer" containerID="027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"
Oct 30 12:20:26.581139 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:26.581042    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff\": container with ID starting with 027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff not found: ID does not exist" containerID="027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"
Oct 30 12:20:26.581139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.581087    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff"} err="failed to get container status \"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff\": rpc error: code = NotFound desc = could not find container \"027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff\": container with ID starting with 027944d23fbc88a7884ea268f25acfda21d6b6592d4a388a37d5a528e67dc5ff not found: ID does not exist"
Oct 30 12:20:26.582915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.582891    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4f7443/payload-processing-external-model-e2e-extmod-a-4f7443-54f4zjtrg"]
Oct 30 12:20:26.903062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.903023    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:26.904708 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.904670    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:26.923537 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.923504    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" path="/var/lib/kubelet/pods/f5eac023-f89d-4c9b-bcdd-f75c0bf10f33/volumes"
Oct 30 12:20:26.924104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:26.924085    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t"
Oct 30 12:20:28.352938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.352892    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:20:28.664973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.664930    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:20:28.665655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.665634    4453 cpu_manager.go:414] "containerMap: removing container" podUID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" containerName="payload-processing"
Oct 30 12:20:28.665655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.665654    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" containerName="payload-processing"
Oct 30 12:20:28.665849 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.665775    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5eac023-f89d-4c9b-bcdd-f75c0bf10f33" containerName="payload-processing"
Oct 30 12:20:28.670741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.670723    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.683066 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.683044    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-737f64-openshift-default-dockercfg-4256n\""
Oct 30 12:20:28.700221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.700195    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:20:28.737945 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.737907    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.737996    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738076    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738110    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738150    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738174    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738223    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738359    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.738420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.738404    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfqw\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.839898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.839858    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840167    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840214    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840246    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840265    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840303    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.840438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840412    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.841886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840463    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfqw\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.841886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.840625    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.841886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.841598    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.841886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.841878    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.842864 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.842125    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.843291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.843261    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.843438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.843366    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.843556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.843530    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.843773 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.843751    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.862902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.862877    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.863420 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.863403    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfqw\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\") pod \"e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") " pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:28.985394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:28.985297    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:29.327565 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:29.327533    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c9c68b_2298_4fc4_9d7f_55b32f86274e.slice/crio-cf1d897efb0dfa0adfd071b9bb059576eeeaceb254c15bf3517f79e2a37f65d1 WatchSource:0}: Error finding container cf1d897efb0dfa0adfd071b9bb059576eeeaceb254c15bf3517f79e2a37f65d1: Status 404 returned error can't find the container with id cf1d897efb0dfa0adfd071b9bb059576eeeaceb254c15bf3517f79e2a37f65d1
Oct 30 12:20:29.328770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.328716    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:20:29.331132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.331088    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:29.331228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.331188    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:29.331277 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.331235    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:20:29.528590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.528555    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" event={"ID":"b2c9c68b-2298-4fc4-9d7f-55b32f86274e","Type":"ContainerStarted","Data":"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"}
Oct 30 12:20:29.529079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.528617    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" event={"ID":"b2c9c68b-2298-4fc4-9d7f-55b32f86274e","Type":"ContainerStarted","Data":"cf1d897efb0dfa0adfd071b9bb059576eeeaceb254c15bf3517f79e2a37f65d1"}
Oct 30 12:20:29.568714 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.568650    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podStartSLOduration=1.568634096 podStartE2EDuration="1.568634096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:29.567215875 +0000 UTC m=+3028.787500786" watchObservedRunningTime="2026-10-30 12:20:29.568634096 +0000 UTC m=+3028.788919047"
Oct 30 12:20:29.985398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:29.985369    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:30.988227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:30.988190    4453 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 12:20:30.988743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:30.988243    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 12:20:31.985734 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:31.985693    4453 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 12:20:31.985952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:31.985765    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 12:20:32.986056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:32.986020    4453 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 12:20:32.986451 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:32.986091    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 12:20:33.986412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:33.986373    4453 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 12:20:33.986902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:33.986435    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 12:20:34.986249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:34.986214    4453 patch_prober.go:29] interesting pod/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 12:20:34.986446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:34.986261    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 12:20:35.991241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:35.991210    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:35.991705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:35.991686    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:35.992344 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:35.992316    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:20:37.572266 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:20:37.572226    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:20:40.740994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.740957    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:20:40.746228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.746207    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.761883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.761856    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a1f322-dockercfg-dz286\""
Oct 30 12:20:40.775254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.775218    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:20:40.830415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.830376    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a1f322\""
Oct 30 12:20:40.871068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.871028    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbmg\" (UniqueName: \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.871269 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.871101    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.972840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.972796    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbmg\" (UniqueName: \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.973015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.972855    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.973636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.973613    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:40.994210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:40.994128    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbmg\" (UniqueName: \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\") pod \"payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:41.062556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.062528    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:41.369986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.369953    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:20:41.370716 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:20:41.370691    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c911504_d363_4f1f_a526_3f98f4810061.slice/crio-39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817 WatchSource:0}: Error finding container 39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817: Status 404 returned error can't find the container with id 39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817
Oct 30 12:20:41.617443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.617325    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" event={"ID":"3c911504-d363-4f1f-a526-3f98f4810061","Type":"ContainerStarted","Data":"961fe6606bb4d7467ad23b1010be8db88488bca05cd936473f8ccfc6b2b23ec9"}
Oct 30 12:20:41.617443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.617404    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" event={"ID":"3c911504-d363-4f1f-a526-3f98f4810061","Type":"ContainerStarted","Data":"39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817"}
Oct 30 12:20:41.617443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.617433    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:41.654260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:41.654209    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" podStartSLOduration=1.654192487 podStartE2EDuration="1.654192487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:20:41.652827418 +0000 UTC m=+3040.873112502" watchObservedRunningTime="2026-10-30 12:20:41.654192487 +0000 UTC m=+3040.874477397"
Oct 30 12:20:52.684381 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:52.673265    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:20:59.735636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.735587    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:20:59.742444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.742400    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:20:59.767125 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.767097    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5cf206-dockercfg-prp7k\""
Oct 30 12:20:59.769820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.769779    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5cf206\""
Oct 30 12:20:59.771802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.771778    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:20:59.797784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.797457    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gfh\" (UniqueName: \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:20:59.797784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.797732    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:20:59.960355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.946109    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:20:59.960355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.946247    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gfh\" (UniqueName: \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:20:59.960355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:20:59.952493    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:21:00.000820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.000741    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gfh\" (UniqueName: \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\") pod \"payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:21:00.065218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.065181    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:21:00.377460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.377433    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:21:00.378839 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:21:00.378813    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87cda50b_fdb2_4b11_913e_544644360e39.slice/crio-ee1a79235ad34c5dbc940721fb035f3532d6690b3f9dfc06d380d94bedfbb9df WatchSource:0}: Error finding container ee1a79235ad34c5dbc940721fb035f3532d6690b3f9dfc06d380d94bedfbb9df: Status 404 returned error can't find the container with id ee1a79235ad34c5dbc940721fb035f3532d6690b3f9dfc06d380d94bedfbb9df
Oct 30 12:21:00.772027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.771924    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" event={"ID":"87cda50b-fdb2-4b11-913e-544644360e39","Type":"ContainerStarted","Data":"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"}
Oct 30 12:21:00.772027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.771990    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" event={"ID":"87cda50b-fdb2-4b11-913e-544644360e39","Type":"ContainerStarted","Data":"ee1a79235ad34c5dbc940721fb035f3532d6690b3f9dfc06d380d94bedfbb9df"}
Oct 30 12:21:00.772027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.772014    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:21:00.807915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:00.807861    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" podStartSLOduration=1.807846477 podStartE2EDuration="1.807846477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:00.806196771 +0000 UTC m=+3060.026481678" watchObservedRunningTime="2026-10-30 12:21:00.807846477 +0000 UTC m=+3060.028131424"
Oct 30 12:21:01.667546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.667500    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:21:01.672121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.672094    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.679564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679532    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.679731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679675    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.679812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679774    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.679812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679803    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.679929 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679844    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.680276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679933    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.680276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.679959    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dz5\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.680276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.680030    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.680276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.680088    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.703563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.703527    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:21:01.781784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.781752    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.781858    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.781928    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.781961    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.781999    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782035    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782060    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dz5\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782086    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782148    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782372    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782455    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782487    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.782853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.782832    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.783133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.783108    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.784382 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.784361    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.784530 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.784514    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.802840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.802808    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.803030 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.803009    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dz5\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2924z\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:01.987176 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:01.987083    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:02.307963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.307932    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:21:02.308980 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:21:02.308950    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768b7c4b_6c3f_46bc_bab4_350c558127c2.slice/crio-f1f24125685a869a275e379fee9623ebd54a753be4740dea37b3b3891b124b03 WatchSource:0}: Error finding container f1f24125685a869a275e379fee9623ebd54a753be4740dea37b3b3891b124b03: Status 404 returned error can't find the container with id f1f24125685a869a275e379fee9623ebd54a753be4740dea37b3b3891b124b03
Oct 30 12:21:02.311894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.311861    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:21:02.311967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.311930    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:21:02.311967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.311960    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:21:02.790553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.790514    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" event={"ID":"768b7c4b-6c3f-46bc-bab4-350c558127c2","Type":"ContainerStarted","Data":"69861f7662cd4c21b00bf285272225d5a6f4f9edc083d8f8aeac879a8ac5712a"}
Oct 30 12:21:02.791037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.790568    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" event={"ID":"768b7c4b-6c3f-46bc-bab4-350c558127c2","Type":"ContainerStarted","Data":"f1f24125685a869a275e379fee9623ebd54a753be4740dea37b3b3891b124b03"}
Oct 30 12:21:02.840533 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.840477    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podStartSLOduration=1.8404599259999999 podStartE2EDuration="1.840459926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:21:02.839101424 +0000 UTC m=+3062.059386338" watchObservedRunningTime="2026-10-30 12:21:02.840459926 +0000 UTC m=+3062.060744836"
Oct 30 12:21:02.987910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:02.987878    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:04.008356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:03.989652    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:04.008356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:03.989739    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:04.988252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:04.988221    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:04.988464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:04.988270    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:05.987824 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:05.987787    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:05.988213 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:05.987839    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:06.987571 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:06.987536    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:06.987777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:06.987592    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:07.571803 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:21:07.571771    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:21:07.987800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:07.987766    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:07.987977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:07.987814    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:08.987616 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:08.987581    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 12:21:08.988009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:08.987634    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 12:21:10.823554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:10.823511    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:10.824376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:10.824351    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:10.824958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:10.824937    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:21:11.782309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:21:11.782279    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:21:37.572138 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:21:37.572099    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:22:07.572485 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:07.572448    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:22:22.566224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.566183    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:22:22.570899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.570882    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.594379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.594354    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9b2407\""
Oct 30 12:22:22.594507 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.594377    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9b2407-dockercfg-fvsvn\""
Oct 30 12:22:22.598171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.598146    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:22:22.680368 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.653957    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:22:22.692208 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.692184    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.708379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.708322    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.708593 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.708559    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zn9c\" (UniqueName: \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.726464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.726414    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:22:22.809704 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.809670    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.809863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.809729    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zn9c\" (UniqueName: \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.809966 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.809940    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.810046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.810032    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhgh\" (UniqueName: \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.810945 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.810919    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.831878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.831791    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zn9c\" (UniqueName: \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\") pod \"payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.885984 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.885955    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:22.911068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.911037    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhgh\" (UniqueName: \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.911209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.911159    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.911932 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.911910    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:22.932221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:22.932194    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhgh\" (UniqueName: \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\") pod \"payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:23.017575 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.017535    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:23.193672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.193647    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:22:23.200246 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:22:23.200210    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20717866_3b47_441f_8deb_e1f810cb9c43.slice/crio-bb28fefe89b61f30eebcb991d823fc3454da8fb1e15143b83c1889518b0f572d WatchSource:0}: Error finding container bb28fefe89b61f30eebcb991d823fc3454da8fb1e15143b83c1889518b0f572d: Status 404 returned error can't find the container with id bb28fefe89b61f30eebcb991d823fc3454da8fb1e15143b83c1889518b0f572d
Oct 30 12:22:23.276445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.276397    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"]
Oct 30 12:22:23.281678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.281658    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.307423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.304315    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"routing-overlay\""
Oct 30 12:22:23.308278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.308251    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"]
Oct 30 12:22:23.308694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.308676    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-plugins-e2e-extmod-a-737f64\""
Oct 30 12:22:23.308694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.308682    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"agc-cred-93e88bcdc5ff\""
Oct 30 12:22:23.308986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.308919    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"openshift-service-ca.crt\""
Oct 30 12:22:23.310598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.310579    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"payload-processing-external-model-e2e--f69e4050-dockercfg-s2k9p\""
Oct 30 12:22:23.310698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.310658    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-737f64\"/\"kube-root-ca.crt\""
Oct 30 12:22:23.328953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.328775    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:22:23.333198 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:22:23.333171    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e50b525_461a_40a4_aa8e_e5bafaa95f6e.slice/crio-d5e347e21bd1dda4eabd53109d20ce1601ff25e99a227df02f58a11f75fe7ea3 WatchSource:0}: Error finding container d5e347e21bd1dda4eabd53109d20ce1601ff25e99a227df02f58a11f75fe7ea3: Status 404 returned error can't find the container with id d5e347e21bd1dda4eabd53109d20ce1601ff25e99a227df02f58a11f75fe7ea3
Oct 30 12:22:23.390233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.389239    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" event={"ID":"20717866-3b47-441f-8deb-e1f810cb9c43","Type":"ContainerStarted","Data":"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"}
Oct 30 12:22:23.390233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.389310    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:23.390233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.389345    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" event={"ID":"20717866-3b47-441f-8deb-e1f810cb9c43","Type":"ContainerStarted","Data":"bb28fefe89b61f30eebcb991d823fc3454da8fb1e15143b83c1889518b0f572d"}
Oct 30 12:22:23.390745 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.390720    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" event={"ID":"8e50b525-461a-40a4-aa8e-e5bafaa95f6e","Type":"ContainerStarted","Data":"d5e347e21bd1dda4eabd53109d20ce1601ff25e99a227df02f58a11f75fe7ea3"}
Oct 30 12:22:23.420289 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.420258    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.420434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.420412    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6k9x\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.420620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.420603    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.420685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.420637    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.429546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.429499    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" podStartSLOduration=1.429486392 podStartE2EDuration="1.429486392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:23.425937295 +0000 UTC m=+3142.646222205" watchObservedRunningTime="2026-10-30 12:22:23.429486392 +0000 UTC m=+3142.649771343"
Oct 30 12:22:23.522157 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.522122    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.522297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.522171    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.522297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.522212    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.522297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.522275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6k9x\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.522530 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:23.522340    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-737f64/agc-cred-93e88bcdc5ff: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:23.522530 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:23.522361    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:23.522792 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:23.522773    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials podName:0cdc510e-4d62-4817-b895-0f461a7ac3bf nodeName:}" failed. No retries permitted until 2026-10-30 12:22:24.022746935 +0000 UTC m=+3143.243031850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf") : secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:23.523117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.523097    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.523198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.523172    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:23.545948 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:23.545924    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6k9x\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:24.029632 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:24.029540    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:24.030054 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:24.029649    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-737f64/agc-cred-93e88bcdc5ff: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:24.030054 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:24.029666    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:24.030054 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:24.029746    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials podName:0cdc510e-4d62-4817-b895-0f461a7ac3bf nodeName:}" failed. No retries permitted until 2026-10-30 12:22:25.029728605 +0000 UTC m=+3144.250013496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf") : secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:24.399177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:24.399141    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" event={"ID":"8e50b525-461a-40a4-aa8e-e5bafaa95f6e","Type":"ContainerStarted","Data":"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"}
Oct 30 12:22:24.399395 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:24.399265    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:24.437382 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:24.437306    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" podStartSLOduration=2.437293303 podStartE2EDuration="2.437293303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:24.43336986 +0000 UTC m=+3143.653654767" watchObservedRunningTime="2026-10-30 12:22:24.437293303 +0000 UTC m=+3143.657578211"
Oct 30 12:22:24.664295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:24.664195    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"]
Oct 30 12:22:24.664706 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:24.664677    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d" podUID="0cdc510e-4d62-4817-b895-0f461a7ac3bf"
Oct 30 12:22:25.046409 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.045578    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") " pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:25.046409 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:25.045785    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-737f64/agc-cred-93e88bcdc5ff: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:25.046409 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:25.045801    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d: secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:25.046409 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:25.045876    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials podName:0cdc510e-4d62-4817-b895-0f461a7ac3bf nodeName:}" failed. No retries permitted until 2026-10-30 12:22:27.045855867 +0000 UTC m=+3146.266140779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf") : secret "agc-cred-93e88bcdc5ff" not found
Oct 30 12:22:25.406504 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.406475    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:25.416360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.416313    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:25.449246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449213    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\") pod \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") "
Oct 30 12:22:25.449433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449295    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\") pod \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") "
Oct 30 12:22:25.449433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449379    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\") pod \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\" (UID: \"0cdc510e-4d62-4817-b895-0f461a7ac3bf\") "
Oct 30 12:22:25.449688 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449666    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume" pod "0cdc510e-4d62-4817-b895-0f461a7ac3bf" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:25.449742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449663    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume" pod "0cdc510e-4d62-4817-b895-0f461a7ac3bf" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:25.449835 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449822    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:25.449894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.449837    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0cdc510e-4d62-4817-b895-0f461a7ac3bf-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:25.451634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.451611    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x" pod "0cdc510e-4d62-4817-b895-0f461a7ac3bf" (UID: "0cdc510e-4d62-4817-b895-0f461a7ac3bf"). InnerVolumeSpecName "kube-api-access-d6k9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:25.550975 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:25.550932    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6k9x\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-kube-api-access-d6k9x\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:26.413584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:26.413551    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"
Oct 30 12:22:26.483432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:26.483398    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"]
Oct 30 12:22:26.488179 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:26.488158    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-737f64/payload-processing-external-model-e2e-extmod-a-737f64-bb97d5w4d"]
Oct 30 12:22:26.564450 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:26.564418    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0cdc510e-4d62-4817-b895-0f461a7ac3bf-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:26.912892 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:26.912857    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdc510e-4d62-4817-b895-0f461a7ac3bf" path="/var/lib/kubelet/pods/0cdc510e-4d62-4817-b895-0f461a7ac3bf/volumes"
Oct 30 12:22:34.403204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:34.403170    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:22:35.410523 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:35.410487    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:22:37.572535 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:37.572476    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:22:39.600635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.600572    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:39.605588 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.605567    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.618042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.617986    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 12:22:39.627455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.627422    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c6bdc992808c\""
Oct 30 12:22:39.627668 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.627421    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 12:22:39.627668 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.627446    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 12:22:39.627867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.627458    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 12:22:39.627867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.627531    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l2vmt\""
Oct 30 12:22:39.632852 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.632823    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:39.701846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.701803    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr9wj\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.702035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.701853    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.702035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.701986    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.702155 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.702084    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.828356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.808995    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.828356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.809521    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr9wj\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.828356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.809573    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.828356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.810020    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.828356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.825265    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.874234 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.874196    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.941843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.941801    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:39.943468 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:39.943437    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr9wj\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\") pod \"payload-processing-external-model-8677757f86-6cj5s\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:40.221790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:40.221697    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:40.536611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:40.536547    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:40.536758 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:22:40.536659    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ce09d9_4c3e_4001_b30b_b53786997d6c.slice/crio-5512e59b13600a3d1316b48a69e3183c6d5182350d14da4cd34446e026e8b1dd WatchSource:0}: Error finding container 5512e59b13600a3d1316b48a69e3183c6d5182350d14da4cd34446e026e8b1dd: Status 404 returned error can't find the container with id 5512e59b13600a3d1316b48a69e3183c6d5182350d14da4cd34446e026e8b1dd
Oct 30 12:22:41.548214 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:41.548165    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" event={"ID":"a1ce09d9-4c3e-4001-b30b-b53786997d6c","Type":"ContainerStarted","Data":"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"}
Oct 30 12:22:41.548744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:41.548241    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" event={"ID":"a1ce09d9-4c3e-4001-b30b-b53786997d6c","Type":"ContainerStarted","Data":"5512e59b13600a3d1316b48a69e3183c6d5182350d14da4cd34446e026e8b1dd"}
Oct 30 12:22:41.548744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:41.548274    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:41.588045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:41.587985    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" podStartSLOduration=2.58796737 podStartE2EDuration="2.58796737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:41.58493201 +0000 UTC m=+3160.805216920" watchObservedRunningTime="2026-10-30 12:22:41.58796737 +0000 UTC m=+3160.808252280"
Oct 30 12:22:41.999908 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:41.999876    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:22:42.010448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.010415    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.023883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.023828    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9d71fa\""
Oct 30 12:22:42.024148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.024118    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9d71fa-dockercfg-4p9vt\""
Oct 30 12:22:42.028143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.028117    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:22:42.151355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.140751    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.151715 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.151679    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqk8z\" (UniqueName: \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.260868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.258591    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.260868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.258673    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqk8z\" (UniqueName: \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.260868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.260092    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.291774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.291478    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqk8z\" (UniqueName: \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\") pod \"payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.331977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.331946    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:42.638508 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:42.638484    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:22:42.639716 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:22:42.639692    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7cecb5_1996_4ce0_92f2_21ac8c2339d8.slice/crio-91ead7be2930a05a491ab6e39acc7915261daa0275e155334c2424d51700d7a9 WatchSource:0}: Error finding container 91ead7be2930a05a491ab6e39acc7915261daa0275e155334c2424d51700d7a9: Status 404 returned error can't find the container with id 91ead7be2930a05a491ab6e39acc7915261daa0275e155334c2424d51700d7a9
Oct 30 12:22:43.570531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:43.570478    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" event={"ID":"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8","Type":"ContainerStarted","Data":"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"}
Oct 30 12:22:43.570723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:43.570569    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:43.570723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:43.570589    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" event={"ID":"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8","Type":"ContainerStarted","Data":"91ead7be2930a05a491ab6e39acc7915261daa0275e155334c2424d51700d7a9"}
Oct 30 12:22:43.610680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:43.610618    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" podStartSLOduration=2.6105997370000003 podStartE2EDuration="2.610599737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:43.605219981 +0000 UTC m=+3162.825504902" watchObservedRunningTime="2026-10-30 12:22:43.610599737 +0000 UTC m=+3162.830884649"
Oct 30 12:22:44.165387 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:44.165347    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:46.592775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.592743    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:46.597293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.597272    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.618865 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.618840    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-flw5j\""
Oct 30 12:22:46.624095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.624069    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:46.781353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.781183    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.781353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.781313    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.781614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.781501    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtmn\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.781614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.781570    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.883314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.883268    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.883555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.883511    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtmn\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.883613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.883599    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.883697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.883681    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.884416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.884357    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.884618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.884593    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.887499 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.887474    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.905269 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.905243    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtmn\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\") pod \"payload-processing-external-model-8677757f86-88dkv\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") " pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:46.911406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:46.911373    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:47.248435 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:47.235201    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:47.255909 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:22:47.255877    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e7e220_eb99_4fb6_843e_da7ba860598a.slice/crio-1b685cbc735c71da77caac8543dff8de2c7fde13d21d71bf992856d5708e340b WatchSource:0}: Error finding container 1b685cbc735c71da77caac8543dff8de2c7fde13d21d71bf992856d5708e340b: Status 404 returned error can't find the container with id 1b685cbc735c71da77caac8543dff8de2c7fde13d21d71bf992856d5708e340b
Oct 30 12:22:47.646478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:47.646432    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" event={"ID":"87e7e220-eb99-4fb6-843e-da7ba860598a","Type":"ContainerStarted","Data":"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"}
Oct 30 12:22:47.646878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:47.646507    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" event={"ID":"87e7e220-eb99-4fb6-843e-da7ba860598a","Type":"ContainerStarted","Data":"1b685cbc735c71da77caac8543dff8de2c7fde13d21d71bf992856d5708e340b"}
Oct 30 12:22:47.646878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:47.646566    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:47.685001 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:47.684941    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" podStartSLOduration=1.684924802 podStartE2EDuration="1.684924802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:22:47.681518692 +0000 UTC m=+3166.901803601" watchObservedRunningTime="2026-10-30 12:22:47.684924802 +0000 UTC m=+3166.905209772"
Oct 30 12:22:49.166265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.166202    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" podUID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" containerName="payload-processing" containerID="cri-o://1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f" gracePeriod=25
Oct 30 12:22:49.448017 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.447989    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:49.622688 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.622646    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\") pod \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") "
Oct 30 12:22:49.622887 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.622721    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\") pod \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") "
Oct 30 12:22:49.622887 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.622758    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\") pod \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") "
Oct 30 12:22:49.623001 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.622899    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\") pod \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\" (UID: \"a1ce09d9-4c3e-4001-b30b-b53786997d6c\") "
Oct 30 12:22:49.623158 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.623085    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume" pod "a1ce09d9-4c3e-4001-b30b-b53786997d6c" (UID: "a1ce09d9-4c3e-4001-b30b-b53786997d6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:49.623379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.623312    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume" pod "a1ce09d9-4c3e-4001-b30b-b53786997d6c" (UID: "a1ce09d9-4c3e-4001-b30b-b53786997d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:49.623637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.623617    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:49.623702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.623641    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ce09d9-4c3e-4001-b30b-b53786997d6c-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:49.624971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.624946    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj" pod "a1ce09d9-4c3e-4001-b30b-b53786997d6c" (UID: "a1ce09d9-4c3e-4001-b30b-b53786997d6c"). InnerVolumeSpecName "kube-api-access-jr9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:49.625063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.624966    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials" pod "a1ce09d9-4c3e-4001-b30b-b53786997d6c" (UID: "a1ce09d9-4c3e-4001-b30b-b53786997d6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:49.661490 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.661459    4453 generic.go:356] "Generic (PLEG): container finished" podID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" containerID="1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f" exitCode=0
Oct 30 12:22:49.661614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.661506    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"
Oct 30 12:22:49.661614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.661543    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" event={"ID":"a1ce09d9-4c3e-4001-b30b-b53786997d6c","Type":"ContainerDied","Data":"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"}
Oct 30 12:22:49.661614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.661600    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-6cj5s" event={"ID":"a1ce09d9-4c3e-4001-b30b-b53786997d6c","Type":"ContainerDied","Data":"5512e59b13600a3d1316b48a69e3183c6d5182350d14da4cd34446e026e8b1dd"}
Oct 30 12:22:49.661741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.661628    4453 scope.go:122] "RemoveContainer" containerID="1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"
Oct 30 12:22:49.672098 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.672077    4453 scope.go:122] "RemoveContainer" containerID="1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"
Oct 30 12:22:49.672369 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:49.672345    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f\": container with ID starting with 1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f not found: ID does not exist" containerID="1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"
Oct 30 12:22:49.672452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.672378    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f"} err="failed to get container status \"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f\": rpc error: code = NotFound desc = could not find container \"1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f\": container with ID starting with 1090b48139ca3a1a472f0baad52a3ed0c309e0364e6859b20bc80e1562ec7f8f not found: ID does not exist"
Oct 30 12:22:49.705721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.705660    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:49.711700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.711678    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-6cj5s"]
Oct 30 12:22:49.724968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.724943    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr9wj\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-kube-api-access-jr9wj\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:49.724968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:49.724967    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1ce09d9-4c3e-4001-b30b-b53786997d6c-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:50.912983 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:50.912950    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" path="/var/lib/kubelet/pods/a1ce09d9-4c3e-4001-b30b-b53786997d6c/volumes"
Oct 30 12:22:51.864274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:51.864234    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:54.603387 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:54.599753    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:22:56.865355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:56.865185    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" podUID="87e7e220-eb99-4fb6-843e-da7ba860598a" containerName="payload-processing" containerID="cri-o://2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1" gracePeriod=25
Oct 30 12:22:56.868386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:56.867426    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2924z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:22:56.868386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:56.867503    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:22:57.688209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.688182    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:57.769414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.768178    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\") pod \"87e7e220-eb99-4fb6-843e-da7ba860598a\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") "
Oct 30 12:22:57.769414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.768235    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\") pod \"87e7e220-eb99-4fb6-843e-da7ba860598a\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") "
Oct 30 12:22:57.769414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.768271    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\") pod \"87e7e220-eb99-4fb6-843e-da7ba860598a\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") "
Oct 30 12:22:57.769414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.768320    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\") pod \"87e7e220-eb99-4fb6-843e-da7ba860598a\" (UID: \"87e7e220-eb99-4fb6-843e-da7ba860598a\") "
Oct 30 12:22:57.769414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.768762    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume" pod "87e7e220-eb99-4fb6-843e-da7ba860598a" (UID: "87e7e220-eb99-4fb6-843e-da7ba860598a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:57.770239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.769998    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:57.773413 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.772447    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume" pod "87e7e220-eb99-4fb6-843e-da7ba860598a" (UID: "87e7e220-eb99-4fb6-843e-da7ba860598a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:22:57.793403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.783393    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn" pod "87e7e220-eb99-4fb6-843e-da7ba860598a" (UID: "87e7e220-eb99-4fb6-843e-da7ba860598a"). InnerVolumeSpecName "kube-api-access-7dtmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:57.793403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.783482    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials" pod "87e7e220-eb99-4fb6-843e-da7ba860598a" (UID: "87e7e220-eb99-4fb6-843e-da7ba860598a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:22:57.872196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.872157    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtmn\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-kube-api-access-7dtmn\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:57.872196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.872197    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e7e220-eb99-4fb6-843e-da7ba860598a-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:57.872638 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.872212    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e7e220-eb99-4fb6-843e-da7ba860598a-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:22:57.886635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.886602    4453 generic.go:356] "Generic (PLEG): container finished" podID="87e7e220-eb99-4fb6-843e-da7ba860598a" containerID="2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1" exitCode=0
Oct 30 12:22:57.886787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.886654    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv"
Oct 30 12:22:57.886787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.886654    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" event={"ID":"87e7e220-eb99-4fb6-843e-da7ba860598a","Type":"ContainerDied","Data":"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"}
Oct 30 12:22:57.886864 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.886811    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8677757f86-88dkv" event={"ID":"87e7e220-eb99-4fb6-843e-da7ba860598a","Type":"ContainerDied","Data":"1b685cbc735c71da77caac8543dff8de2c7fde13d21d71bf992856d5708e340b"}
Oct 30 12:22:57.886864 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.886840    4453 scope.go:122] "RemoveContainer" containerID="2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"
Oct 30 12:22:57.898583 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.898564    4453 scope.go:122] "RemoveContainer" containerID="2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"
Oct 30 12:22:57.898863 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:22:57.898844    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1\": container with ID starting with 2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1 not found: ID does not exist" containerID="2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"
Oct 30 12:22:57.898941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.898872    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1"} err="failed to get container status \"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1\": rpc error: code = NotFound desc = could not find container \"2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1\": container with ID starting with 2eea72510dc8b13d2420eac9859d686ea44e95dd8b9d4a55bb616f16d79a27b1 not found: ID does not exist"
Oct 30 12:22:57.932118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.932056    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:57.939896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.939829    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:22:57.940041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.939910    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:22:57.940847 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:57.940825    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8677757f86-88dkv"]
Oct 30 12:22:58.957018 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:22:58.956974    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e7e220-eb99-4fb6-843e-da7ba860598a" path="/var/lib/kubelet/pods/87e7e220-eb99-4fb6-843e-da7ba860598a/volumes"
Oct 30 12:23:00.288925 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.288894    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:23:00.289776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289755    4453 cpu_manager.go:414] "containerMap: removing container" podUID="87e7e220-eb99-4fb6-843e-da7ba860598a" containerName="payload-processing"
Oct 30 12:23:00.289858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289776    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e7e220-eb99-4fb6-843e-da7ba860598a" containerName="payload-processing"
Oct 30 12:23:00.289858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289792    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" containerName="payload-processing"
Oct 30 12:23:00.289858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289800    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" containerName="payload-processing"
Oct 30 12:23:00.289993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289924    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e7e220-eb99-4fb6-843e-da7ba860598a" containerName="payload-processing"
Oct 30 12:23:00.289993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.289932    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ce09d9-4c3e-4001-b30b-b53786997d6c" containerName="payload-processing"
Oct 30 12:23:00.295104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.295084    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.316771 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.316743    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b15ec3\"/\"agc-cred-d93c626e8305\""
Oct 30 12:23:00.320606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.320577    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:23:00.349099 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.349055    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.349272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.349242    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.349372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.349305    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.349535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.349514    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbstk\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.450781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.450746    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.450981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.450825    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.450981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.450861    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.450981 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.450932    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbstk\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.451716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.451691    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.451833 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.451732    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.453554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.453532    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.471324 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.471304    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbstk\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.609536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.609510    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:00.914237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.914212    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:23:00.915145 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:00.915122    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f225d10_33d5_459f_907a_020e3c053bd3.slice/crio-322215ccd2b87bb48af8fa83bfce83974e9af8c79221d43638e3378df7acb34d WatchSource:0}: Error finding container 322215ccd2b87bb48af8fa83bfce83974e9af8c79221d43638e3378df7acb34d: Status 404 returned error can't find the container with id 322215ccd2b87bb48af8fa83bfce83974e9af8c79221d43638e3378df7acb34d
Oct 30 12:23:00.962537 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:00.961326    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" event={"ID":"1f225d10-33d5-459f-907a-020e3c053bd3","Type":"ContainerStarted","Data":"322215ccd2b87bb48af8fa83bfce83974e9af8c79221d43638e3378df7acb34d"}
Oct 30 12:23:01.376360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.376304    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:23:01.384191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.384152    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.401507 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.401467    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ae3538\""
Oct 30 12:23:01.402566 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.402537    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:23:01.464694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.464655    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.464878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.464805    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2t2\" (UniqueName: \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.592355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.590388    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.592355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.590500    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2t2\" (UniqueName: \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.592355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.591806    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.611973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.611941    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2t2\" (UniqueName: \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\") pod \"payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.702233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.702125    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:01.970217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.970115    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" event={"ID":"1f225d10-33d5-459f-907a-020e3c053bd3","Type":"ContainerStarted","Data":"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"}
Oct 30 12:23:01.970407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:01.970349    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:02.009168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.009112    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" podStartSLOduration=2.009099545 podStartE2EDuration="2.009099545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:02.008123423 +0000 UTC m=+3181.228408334" watchObservedRunningTime="2026-10-30 12:23:02.009099545 +0000 UTC m=+3181.229384486"
Oct 30 12:23:02.036555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.036520    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:23:02.037117 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:02.037087    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acacdad_31aa_4fdc_aea1_83197367f134.slice/crio-121b24a82ffadd0282d460c46e04cd81bd9869e404a1e631577653e821bc5736 WatchSource:0}: Error finding container 121b24a82ffadd0282d460c46e04cd81bd9869e404a1e631577653e821bc5736: Status 404 returned error can't find the container with id 121b24a82ffadd0282d460c46e04cd81bd9869e404a1e631577653e821bc5736
Oct 30 12:23:02.160118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.160085    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:23:02.164816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.164799    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.176844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.176823    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-59b8a9\"/\"agc-cred-5d59545765d0\""
Oct 30 12:23:02.184744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.184720    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:23:02.312570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.312482    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.312726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.312639    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnvw\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.312726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.312705    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.312823 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.312788    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.414366 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.414310    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.414918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.414494    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.414918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.414569    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnvw\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.414918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.414600    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.415186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.415159    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.415364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.415322    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.417145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.417128    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.435595 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.435571    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnvw\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.481249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.481220    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:02.785173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.785140    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:23:02.785388 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:02.785362    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36351911_4cda_4385_8a86_d72116234201.slice/crio-26e86d5a6edc624a8f3b8591a6153a4ebf66ff5cbfc5699d8ef07a159a59628e WatchSource:0}: Error finding container 26e86d5a6edc624a8f3b8591a6153a4ebf66ff5cbfc5699d8ef07a159a59628e: Status 404 returned error can't find the container with id 26e86d5a6edc624a8f3b8591a6153a4ebf66ff5cbfc5699d8ef07a159a59628e
Oct 30 12:23:02.978788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.978751    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" event={"ID":"1acacdad-31aa-4fdc-aea1-83197367f134","Type":"ContainerStarted","Data":"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"}
Oct 30 12:23:02.978972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.978806    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" event={"ID":"1acacdad-31aa-4fdc-aea1-83197367f134","Type":"ContainerStarted","Data":"121b24a82ffadd0282d460c46e04cd81bd9869e404a1e631577653e821bc5736"}
Oct 30 12:23:02.978972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.978830    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:02.980630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.980603    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" event={"ID":"36351911-4cda-4385-8a86-d72116234201","Type":"ContainerStarted","Data":"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"}
Oct 30 12:23:02.980752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.980648    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" event={"ID":"36351911-4cda-4385-8a86-d72116234201","Type":"ContainerStarted","Data":"26e86d5a6edc624a8f3b8591a6153a4ebf66ff5cbfc5699d8ef07a159a59628e"}
Oct 30 12:23:02.980819 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:02.980800    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:03.016411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:03.016366    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" podStartSLOduration=2.016349268 podStartE2EDuration="2.016349268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:03.013443053 +0000 UTC m=+3182.233727962" watchObservedRunningTime="2026-10-30 12:23:03.016349268 +0000 UTC m=+3182.236634178"
Oct 30 12:23:03.049777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:03.049697    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" podStartSLOduration=1.049683599 podStartE2EDuration="1.049683599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:03.04867784 +0000 UTC m=+3182.268962751" watchObservedRunningTime="2026-10-30 12:23:03.049683599 +0000 UTC m=+3182.269968508"
Oct 30 12:23:07.573361 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:07.573297    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:23:12.984388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:12.984356    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:23:13.057547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:13.057515    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:23:13.990350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:13.990267    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:23:13.990797 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:13.990356    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:23:14.101760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:14.101731    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:23:18.058188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.058148    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" podUID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" containerName="payload-processing" containerID="cri-o://f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268" gracePeriod=25
Oct 30 12:23:18.376687 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.376658    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:23:18.469785 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.469752    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\") pod \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") "
Oct 30 12:23:18.469970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.469899    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\") pod \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") "
Oct 30 12:23:18.469970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.469931    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\") pod \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") "
Oct 30 12:23:18.470097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.469976    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\") pod \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\" (UID: \"31e5dd90-5e58-48ab-8ef1-1a357e8a5536\") "
Oct 30 12:23:18.470235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.470134    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume" pod "31e5dd90-5e58-48ab-8ef1-1a357e8a5536" (UID: "31e5dd90-5e58-48ab-8ef1-1a357e8a5536"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:23:18.470415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.470392    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume" pod "31e5dd90-5e58-48ab-8ef1-1a357e8a5536" (UID: "31e5dd90-5e58-48ab-8ef1-1a357e8a5536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:23:18.470714 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.470695    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:18.470775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.470719    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:18.472083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.472061    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials" pod "31e5dd90-5e58-48ab-8ef1-1a357e8a5536" (UID: "31e5dd90-5e58-48ab-8ef1-1a357e8a5536"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:23:18.472175 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.472086    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2" pod "31e5dd90-5e58-48ab-8ef1-1a357e8a5536" (UID: "31e5dd90-5e58-48ab-8ef1-1a357e8a5536"). InnerVolumeSpecName "kube-api-access-f6fv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:23:18.571956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.571916    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:18.571956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:18.571947    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fv2\" (UniqueName: \"kubernetes.io/projected/31e5dd90-5e58-48ab-8ef1-1a357e8a5536-kube-api-access-f6fv2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:19.102053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102012    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" podUID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" containerName="payload-processing" containerID="cri-o://ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04" gracePeriod=25
Oct 30 12:23:19.102620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102380    4453 generic.go:356] "Generic (PLEG): container finished" podID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" containerID="f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268" exitCode=0
Oct 30 12:23:19.102620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102416    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" event={"ID":"31e5dd90-5e58-48ab-8ef1-1a357e8a5536","Type":"ContainerDied","Data":"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"}
Oct 30 12:23:19.102620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102444    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"
Oct 30 12:23:19.102620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102470    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf" event={"ID":"31e5dd90-5e58-48ab-8ef1-1a357e8a5536","Type":"ContainerDied","Data":"57d6eb1c56835c0adcc9c2c69c8ba76ee0730d6ca579b95253991e6d7aa478be"}
Oct 30 12:23:19.102620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.102494    4453 scope.go:122] "RemoveContainer" containerID="f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"
Oct 30 12:23:19.115470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.115451    4453 scope.go:122] "RemoveContainer" containerID="f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"
Oct 30 12:23:19.115774 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:19.115749    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268\": container with ID starting with f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268 not found: ID does not exist" containerID="f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"
Oct 30 12:23:19.115854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.115784    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268"} err="failed to get container status \"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268\": rpc error: code = NotFound desc = could not find container \"f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268\": container with ID starting with f2a4c8e9dc18e4cfee35ad027bad4274d1aebf35aba55fc0b66709e0df985268 not found: ID does not exist"
Oct 30 12:23:19.150022 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.149995    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:23:19.159463 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.159433    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787k4lsf"]
Oct 30 12:23:19.379151 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.379127    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:23:19.486841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.486769    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\") pod \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") "
Oct 30 12:23:19.486987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.486854    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\") pod \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") "
Oct 30 12:23:19.486987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.486931    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\") pod \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") "
Oct 30 12:23:19.487059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.486995    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\") pod \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\" (UID: \"72c69550-a1f7-4ec8-a892-ba95db52e8d6\") "
Oct 30 12:23:19.487296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.487268    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume" pod "72c69550-a1f7-4ec8-a892-ba95db52e8d6" (UID: "72c69550-a1f7-4ec8-a892-ba95db52e8d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:23:19.487423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.487274    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume" pod "72c69550-a1f7-4ec8-a892-ba95db52e8d6" (UID: "72c69550-a1f7-4ec8-a892-ba95db52e8d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:23:19.487787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.487766    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:19.487787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.487787    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72c69550-a1f7-4ec8-a892-ba95db52e8d6-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:19.489194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.489173    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials" pod "72c69550-a1f7-4ec8-a892-ba95db52e8d6" (UID: "72c69550-a1f7-4ec8-a892-ba95db52e8d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:23:19.489299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.489229    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd" pod "72c69550-a1f7-4ec8-a892-ba95db52e8d6" (UID: "72c69550-a1f7-4ec8-a892-ba95db52e8d6"). InnerVolumeSpecName "kube-api-access-9m8cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:23:19.588845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.588810    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:19.588845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:19.588840    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8cd\" (UniqueName: \"kubernetes.io/projected/72c69550-a1f7-4ec8-a892-ba95db52e8d6-kube-api-access-9m8cd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:23:20.112016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.111974    4453 generic.go:356] "Generic (PLEG): container finished" podID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" containerID="ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04" exitCode=0
Oct 30 12:23:20.112547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.112029    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"
Oct 30 12:23:20.112547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.112061    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" event={"ID":"72c69550-a1f7-4ec8-a892-ba95db52e8d6","Type":"ContainerDied","Data":"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"}
Oct 30 12:23:20.112547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.112144    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w" event={"ID":"72c69550-a1f7-4ec8-a892-ba95db52e8d6","Type":"ContainerDied","Data":"c05bf4b1be4d9e0745a976728c3f185e83df5a623db6214c679e7d2f2cd54c57"}
Oct 30 12:23:20.112547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.112175    4453 scope.go:122] "RemoveContainer" containerID="ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"
Oct 30 12:23:20.128541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.128518    4453 scope.go:122] "RemoveContainer" containerID="ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"
Oct 30 12:23:20.130167 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:20.130144    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04\": container with ID starting with ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04 not found: ID does not exist" containerID="ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"
Oct 30 12:23:20.130262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.130175    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04"} err="failed to get container status \"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04\": rpc error: code = NotFound desc = could not find container \"ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04\": container with ID starting with ac42fd65ea21c53eda0990a262e05b67abce4c437026f70f1358c43fa4677d04 not found: ID does not exist"
Oct 30 12:23:20.157866 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.157844    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:23:20.167096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.167074    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9wfr7w"]
Oct 30 12:23:20.913068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.913033    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" path="/var/lib/kubelet/pods/31e5dd90-5e58-48ab-8ef1-1a357e8a5536/volumes"
Oct 30 12:23:20.913539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:20.913522    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" path="/var/lib/kubelet/pods/72c69550-a1f7-4ec8-a892-ba95db52e8d6/volumes"
Oct 30 12:23:30.349193 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.349154    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350018    4453 cpu_manager.go:414] "containerMap: removing container" podUID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" containerName="payload-processing"
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350035    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" containerName="payload-processing"
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350066    4453 cpu_manager.go:414] "containerMap: removing container" podUID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" containerName="payload-processing"
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350074    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" containerName="payload-processing"
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350190    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c69550-a1f7-4ec8-a892-ba95db52e8d6" containerName="payload-processing"
Oct 30 12:23:30.350449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.350207    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e5dd90-5e58-48ab-8ef1-1a357e8a5536" containerName="payload-processing"
Oct 30 12:23:30.355837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.355815    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.380646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.380488    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9d71fa-serving-cert\""
Oct 30 12:23:30.385873 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.385842    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:23:30.454433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.454390    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.454623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.454536    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.454766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.454749    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6kv\" (UniqueName: \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.454833 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.454809    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.559400    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.559676    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6kv\" (UniqueName: \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.559736    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.559896    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.560783    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:30.560901    4453 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-serving-cert: secret "maas-api-e2e-inf-b-9d71fa-serving-cert" not found
Oct 30 12:23:30.561353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:30.560961    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls podName:bfdf710f-22a0-466a-9224-3b209c05f1d4 nodeName:}" failed. No retries permitted until 2026-10-30 12:23:31.06094111 +0000 UTC m=+3210.281226011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls") pod "maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" (UID: "bfdf710f-22a0-466a-9224-3b209c05f1d4") : secret "maas-api-e2e-inf-b-9d71fa-serving-cert" not found
Oct 30 12:23:30.596152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.596122    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6kv\" (UniqueName: \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:30.612661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:30.612632    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:31.071153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:31.071061    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:31.073645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:31.073613    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:31.271354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:31.270379    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:31.630671 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:31.630317    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:23:31.635570 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:31.635539    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf710f_22a0_466a_9224_3b209c05f1d4.slice/crio-753b33968d617b9da92422809031cba290991b2b1f610f5c3914ced5f3332c6d WatchSource:0}: Error finding container 753b33968d617b9da92422809031cba290991b2b1f610f5c3914ced5f3332c6d: Status 404 returned error can't find the container with id 753b33968d617b9da92422809031cba290991b2b1f610f5c3914ced5f3332c6d
Oct 30 12:23:31.636623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:31.636601    4453 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:23:32.248225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:32.248183    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" event={"ID":"bfdf710f-22a0-466a-9224-3b209c05f1d4","Type":"ContainerStarted","Data":"715ddbb8a42aa63854c6b7219af244af423e0ac80927576a8b6c1bf9ec55c5c7"}
Oct 30 12:23:32.248452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:32.248286    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" event={"ID":"bfdf710f-22a0-466a-9224-3b209c05f1d4","Type":"ContainerStarted","Data":"753b33968d617b9da92422809031cba290991b2b1f610f5c3914ced5f3332c6d"}
Oct 30 12:23:32.248452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:32.248369    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:32.290257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:32.290152    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" podStartSLOduration=1.924988541 podStartE2EDuration="2.290133618s" totalImagesPullingTime="365.145077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:32.284528835 +0000 UTC m=+3211.504813746" watchObservedRunningTime="2026-10-30 12:23:32.290133618 +0000 UTC m=+3211.510418531"
Oct 30 12:23:37.572658 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:23:37.572608    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:23:38.262295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:38.262263    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:23:43.463892 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.463858    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:23:43.469771 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.469751    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.484463 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.484425    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-972432cc9b49\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.493941    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-90cb1c6558d0\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.493941    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"agc-cred-5ac20fbbf60f\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.494023    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"kube-root-ca.crt\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.494052    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-e2e--793d1056-dockercfg-zj5q5\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.494083    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.494084    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"openshift-service-ca.crt\""
Oct 30 12:23:43.494139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.494115    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2bc4e2\"/\"routing-overlay\""
Oct 30 12:23:43.500110 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.500086    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:23:43.586674 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.586644    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlrz\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.586827 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.586759    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.586868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.586854    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.586993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.586979    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.688613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.688579    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.688772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.688669    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlrz\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.688772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.688727    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.688894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.688790    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.689253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.689228    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.689628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.689606    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.691432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.691412    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.709778 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.709751    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlrz\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:43.785645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:43.785574    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:44.096706 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:44.096671    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369c5a51_3926_4d61_8d51_26194c9f8988.slice/crio-119f18d4eaafb8e80a308d2d12d2ffbeae06dfa6b93f71b6af15c736877adc60 WatchSource:0}: Error finding container 119f18d4eaafb8e80a308d2d12d2ffbeae06dfa6b93f71b6af15c736877adc60: Status 404 returned error can't find the container with id 119f18d4eaafb8e80a308d2d12d2ffbeae06dfa6b93f71b6af15c736877adc60
Oct 30 12:23:44.098485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.098462    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:23:44.332438 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.332400    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:23:44.337122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.337098    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.340776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.340742    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" event={"ID":"369c5a51-3926-4d61-8d51-26194c9f8988","Type":"ContainerStarted","Data":"ddce7fd1bedb4d183a7e69272f201d7b5beeef73c5159620fa6b337637224001"}
Oct 30 12:23:44.340919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.340800    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" event={"ID":"369c5a51-3926-4d61-8d51-26194c9f8988","Type":"ContainerStarted","Data":"119f18d4eaafb8e80a308d2d12d2ffbeae06dfa6b93f71b6af15c736877adc60"}
Oct 30 12:23:44.340919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.340837    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:44.349840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.349811    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f8f132\"/\"agc-cred-566f41ac1057\""
Oct 30 12:23:44.357552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.357525    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:23:44.408365 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.408297    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" podStartSLOduration=1.408280579 podStartE2EDuration="1.408280579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:44.406406909 +0000 UTC m=+3223.626691819" watchObservedRunningTime="2026-10-30 12:23:44.408280579 +0000 UTC m=+3223.628565500"
Oct 30 12:23:44.500828 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.500789    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.501294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.500869    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.501294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.500926    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.501294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.500956    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949jb\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.602491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.602412    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.602491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.602453    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.602730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.602499    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.602730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.602518    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949jb\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.603249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.603227    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.603379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.603357    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.605023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.605000    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.622988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.622962    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949jb\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.651853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.651829    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:44.958681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:44.958648    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:23:44.958930 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:44.958905    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a7b4f7_f700_4972_9f32_b1b5ecf32f4b.slice/crio-01e46c55d11ba8d71f14ae003ac83294eaa8f174a49cb476e30dc00f1486e644 WatchSource:0}: Error finding container 01e46c55d11ba8d71f14ae003ac83294eaa8f174a49cb476e30dc00f1486e644: Status 404 returned error can't find the container with id 01e46c55d11ba8d71f14ae003ac83294eaa8f174a49cb476e30dc00f1486e644
Oct 30 12:23:45.350270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:45.350178    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" event={"ID":"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b","Type":"ContainerStarted","Data":"37e74ef0ffa6c2bdd173d4a88bf2291a7913af63c146e47c6976d0065febfaba"}
Oct 30 12:23:45.350270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:45.350237    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" event={"ID":"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b","Type":"ContainerStarted","Data":"01e46c55d11ba8d71f14ae003ac83294eaa8f174a49cb476e30dc00f1486e644"}
Oct 30 12:23:45.350526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:45.350277    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:45.388990 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:45.388886    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" podStartSLOduration=1.38886826 podStartE2EDuration="1.38886826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:45.386486002 +0000 UTC m=+3224.606770911" watchObservedRunningTime="2026-10-30 12:23:45.38886826 +0000 UTC m=+3224.609153171"
Oct 30 12:23:50.829589 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.829465    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:23:50.852775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.835926    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.852775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.850884    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ae3538-serving-cert\""
Oct 30 12:23:50.866366 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.865708    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:23:50.882356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.882090    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.882356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.882176    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtkh\" (UniqueName: \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.882577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.882373    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.882577 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.882467    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.983900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.983866    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.984073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.983975    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.984073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.984012    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtkh\" (UniqueName: \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.984220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.984197    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.984768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.984740    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.986623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.986604    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:50.986712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:50.986625    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:51.006036 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:51.006008    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtkh\" (UniqueName: \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\") pod \"maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:51.151412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:51.151376    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:51.459928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:51.459895    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:23:51.460701 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:23:51.460675    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a23799d_4c42_4f57_be2a_21c1067e6e2e.slice/crio-5b48a6d00c25943abcec792ef028d244dcf3fc4bb3032e4144af420be8b7b407 WatchSource:0}: Error finding container 5b48a6d00c25943abcec792ef028d244dcf3fc4bb3032e4144af420be8b7b407: Status 404 returned error can't find the container with id 5b48a6d00c25943abcec792ef028d244dcf3fc4bb3032e4144af420be8b7b407
Oct 30 12:23:52.414504 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:52.414461    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" event={"ID":"0a23799d-4c42-4f57-be2a-21c1067e6e2e","Type":"ContainerStarted","Data":"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"}
Oct 30 12:23:52.414950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:52.414537    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:23:52.414950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:52.414562    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" event={"ID":"0a23799d-4c42-4f57-be2a-21c1067e6e2e","Type":"ContainerStarted","Data":"5b48a6d00c25943abcec792ef028d244dcf3fc4bb3032e4144af420be8b7b407"}
Oct 30 12:23:52.454954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:52.454895    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" podStartSLOduration=2.000447623 podStartE2EDuration="2.45487503s" totalImagesPullingTime="454.427407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:23:52.452052856 +0000 UTC m=+3231.672337794" watchObservedRunningTime="2026-10-30 12:23:52.45487503 +0000 UTC m=+3231.675159939"
Oct 30 12:23:55.353510 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:55.353476    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:23:56.371446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:56.371414    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:23:56.450150 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:56.449322    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:23:58.427693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:23:58.427664    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:24:01.449845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.449803    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" podUID="688e42cb-19e3-4835-97b7-43e7eedcdac0" containerName="payload-processing" containerID="cri-o://56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e" gracePeriod=25
Oct 30 12:24:01.728150 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.728121    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:24:01.866054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866020    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\") pod \"688e42cb-19e3-4835-97b7-43e7eedcdac0\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") "
Oct 30 12:24:01.866214 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866164    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\") pod \"688e42cb-19e3-4835-97b7-43e7eedcdac0\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") "
Oct 30 12:24:01.866257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866241    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\") pod \"688e42cb-19e3-4835-97b7-43e7eedcdac0\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") "
Oct 30 12:24:01.866367 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866354    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\") pod \"688e42cb-19e3-4835-97b7-43e7eedcdac0\" (UID: \"688e42cb-19e3-4835-97b7-43e7eedcdac0\") "
Oct 30 12:24:01.866732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866700    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume" pod "688e42cb-19e3-4835-97b7-43e7eedcdac0" (UID: "688e42cb-19e3-4835-97b7-43e7eedcdac0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:24:01.866854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866710    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume" pod "688e42cb-19e3-4835-97b7-43e7eedcdac0" (UID: "688e42cb-19e3-4835-97b7-43e7eedcdac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:24:01.866899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866886    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:24:01.866935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.866908    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/688e42cb-19e3-4835-97b7-43e7eedcdac0-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:24:01.868369 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.868322    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr" pod "688e42cb-19e3-4835-97b7-43e7eedcdac0" (UID: "688e42cb-19e3-4835-97b7-43e7eedcdac0"). InnerVolumeSpecName "kube-api-access-m2qnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:24:01.868465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.868369    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials" pod "688e42cb-19e3-4835-97b7-43e7eedcdac0" (UID: "688e42cb-19e3-4835-97b7-43e7eedcdac0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:24:01.968898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.968825    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:24:01.968898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:01.968858    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2qnr\" (UniqueName: \"kubernetes.io/projected/688e42cb-19e3-4835-97b7-43e7eedcdac0-kube-api-access-m2qnr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:24:02.501989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.501952    4453 generic.go:356] "Generic (PLEG): container finished" podID="688e42cb-19e3-4835-97b7-43e7eedcdac0" containerID="56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e" exitCode=0
Oct 30 12:24:02.502458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.502002    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"
Oct 30 12:24:02.502458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.502030    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" event={"ID":"688e42cb-19e3-4835-97b7-43e7eedcdac0","Type":"ContainerDied","Data":"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"}
Oct 30 12:24:02.502458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.502082    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2" event={"ID":"688e42cb-19e3-4835-97b7-43e7eedcdac0","Type":"ContainerDied","Data":"21583bd3a37fecc49f8a6977d311c0f0137a112b0e7292be88e979a0d7906a17"}
Oct 30 12:24:02.502458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.502103    4453 scope.go:122] "RemoveContainer" containerID="56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"
Oct 30 12:24:02.513607 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.513591    4453 scope.go:122] "RemoveContainer" containerID="56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"
Oct 30 12:24:02.513900 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:24:02.513872    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e\": container with ID starting with 56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e not found: ID does not exist" containerID="56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"
Oct 30 12:24:02.513952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.513909    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e"} err="failed to get container status \"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e\": rpc error: code = NotFound desc = could not find container \"56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e\": container with ID starting with 56bf1a12d3f3e08c83e0272579805b6791b4bebdaec42d84f011c7f93af8e50e not found: ID does not exist"
Oct 30 12:24:02.547758 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.547730    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:24:02.557257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.557232    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdvdsr2"]
Oct 30 12:24:02.914443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:02.914411    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="688e42cb-19e3-4835-97b7-43e7eedcdac0" path="/var/lib/kubelet/pods/688e42cb-19e3-4835-97b7-43e7eedcdac0/volumes"
Oct 30 12:24:07.572519 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:24:07.572481    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:24:21.358066 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.358029    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:24:21.358831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.358813    4453 cpu_manager.go:414] "containerMap: removing container" podUID="688e42cb-19e3-4835-97b7-43e7eedcdac0" containerName="payload-processing"
Oct 30 12:24:21.358831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.358831    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="688e42cb-19e3-4835-97b7-43e7eedcdac0" containerName="payload-processing"
Oct 30 12:24:21.359000 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.358990    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="688e42cb-19e3-4835-97b7-43e7eedcdac0" containerName="payload-processing"
Oct 30 12:24:21.363917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.363900    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.395215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.395175    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:24:21.454630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454586    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplfn\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.454825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454676    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.454825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454744    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.454825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454776    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.454825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454798    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.455002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.454968    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.455051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.455035    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.455191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.455169    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.455265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.455233    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.457542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.457523    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d71fa-openshift-default-dockercfg-d62s2\""
Oct 30 12:24:21.556856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.556808    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.556909    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557000    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557047    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557113 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557080    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplfn\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557169    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557215    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557283    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557486    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557679 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557662    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557724    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557748    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.557812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.557795    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.558023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.558001    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.559287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.559267    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.559695 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.559674    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.578169 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.578133    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.578268 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.578179    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplfn\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:21.694491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:21.694405    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:22.015026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.014996    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:24:22.015567 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:24:22.015537    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c33f649_f30b_49e6_8093_9a3bd5f9a1fa.slice/crio-40b735b9f07a23dbc08652cf6ef5ac3fc5a1539ef77bfbc098e52c456675720b WatchSource:0}: Error finding container 40b735b9f07a23dbc08652cf6ef5ac3fc5a1539ef77bfbc098e52c456675720b: Status 404 returned error can't find the container with id 40b735b9f07a23dbc08652cf6ef5ac3fc5a1539ef77bfbc098e52c456675720b
Oct 30 12:24:22.019037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.019000    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:22.019165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.019074    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:22.019165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.019106    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:22.667950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.667910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" event={"ID":"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa","Type":"ContainerStarted","Data":"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"}
Oct 30 12:24:22.668385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.667990    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" event={"ID":"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa","Type":"ContainerStarted","Data":"40b735b9f07a23dbc08652cf6ef5ac3fc5a1539ef77bfbc098e52c456675720b"}
Oct 30 12:24:22.694750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.694716    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:22.707466 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:22.707416    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podStartSLOduration=1.707400374 podStartE2EDuration="1.707400374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:22.706809942 +0000 UTC m=+3261.927094846" watchObservedRunningTime="2026-10-30 12:24:22.707400374 +0000 UTC m=+3261.927685285"
Oct 30 12:24:23.730352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:23.729509    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 12:24:23.730352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:23.729574    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 12:24:24.713359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:24.709741    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 12:24:24.713359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:24.709824    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 12:24:25.699372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:25.697364    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 12:24:25.699372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:25.697442    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 12:24:26.713358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:26.710540    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 12:24:26.713358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:26.710612    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 12:24:27.703709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:27.703669    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 12:24:27.703993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:27.703962    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 12:24:28.937904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:28.937851    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:28.938804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:28.938772    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:28.940590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:28.940565    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:24:37.572805 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:24:37.572769    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:24:47.204551 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.204389    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:24:47.217660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.213567    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.228251 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.228224    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0ecae0-openshift-default-dockercfg-q85bv\""
Oct 30 12:24:47.334873 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.334832    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:24:47.335788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.335751    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.336108 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.336082    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.336399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.336313    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.336621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.336601    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.336838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.336820    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.337111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.337081    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.337203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.337130    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.337203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.337188    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.339473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.337300    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfjv\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.438958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.438916    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439014    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439131    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439166    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439192    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439246    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439307    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfjv\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439408    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439514    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439753 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439719    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.439930 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439902    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.440068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.439932    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.440212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.440156    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.440368 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.440319    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.441807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.441786    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.441952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.441934    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.461404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.461298    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfjv\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.462618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.462594    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\") pod \"e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") " pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.565240 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.565205    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:47.951767 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.951735    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:24:47.952057 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:24:47.952031    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9525f0_ff85_42d9_a124_e7c76c4d121f.slice/crio-ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be WatchSource:0}: Error finding container ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be: Status 404 returned error can't find the container with id ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be
Oct 30 12:24:47.954810 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.954774    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:47.954870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.954847    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:47.954907 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:47.954897    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:48.919002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:48.918965    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerStarted","Data":"08d2cd3d089fd401e7dd0b88d204f51aa65945d80704d8d893c04ef288e58846"}
Oct 30 12:24:48.919433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:48.919018    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerStarted","Data":"ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be"}
Oct 30 12:24:48.982733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:48.982682    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podStartSLOduration=1.9826688099999998 podStartE2EDuration="1.98266881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:48.971803846 +0000 UTC m=+3288.192088760" watchObservedRunningTime="2026-10-30 12:24:48.98266881 +0000 UTC m=+3288.202953719"
Oct 30 12:24:49.565729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:49.565696    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:49.571949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:49.571926    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:49.925476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:49.925445    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:49.926446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:49.926422    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:24:51.445580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.445547    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:24:51.450785 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.450764    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.492382 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.492351    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:24:51.612844 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.612800    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.612870    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.612986    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613024    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613123    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613227 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613210    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbm5w\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613241    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613402 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613388    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.613471 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.613412    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714375    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714414    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714461    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714500    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714576    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714614    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714691    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714975 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714819    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbm5w\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.714975 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714859    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.715083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.714977    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.715083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.715058    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.715228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.715210    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.715376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.715314    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.715479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.715377    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.716877 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.716853    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.717026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.716953    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.735728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.735702    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.735848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.735831    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbm5w\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\") pod \"e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") " pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:51.769707 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:51.769683    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:52.110968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:52.110943    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:24:52.113664 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:24:52.113622    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2e39cb_116d_416f_b45a_c55b3ed7ba8a.slice/crio-c0a332911f2d0756ea27d97ef6270e8f992e1b0188398f12a461d50f2f816c93 WatchSource:0}: Error finding container c0a332911f2d0756ea27d97ef6270e8f992e1b0188398f12a461d50f2f816c93: Status 404 returned error can't find the container with id c0a332911f2d0756ea27d97ef6270e8f992e1b0188398f12a461d50f2f816c93
Oct 30 12:24:52.117799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:52.117759    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:52.117901 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:52.117848    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:52.117973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:52.117902    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:24:53.010254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.010205    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" event={"ID":"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a","Type":"ContainerStarted","Data":"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"}
Oct 30 12:24:53.010690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.010286    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" event={"ID":"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a","Type":"ContainerStarted","Data":"c0a332911f2d0756ea27d97ef6270e8f992e1b0188398f12a461d50f2f816c93"}
Oct 30 12:24:53.060071 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.060014    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podStartSLOduration=2.060000848 podStartE2EDuration="2.060000848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:53.059777022 +0000 UTC m=+3292.280061950" watchObservedRunningTime="2026-10-30 12:24:53.060000848 +0000 UTC m=+3292.280285759"
Oct 30 12:24:53.569158 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.569119    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:24:53.577353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.577313    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.619665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.619262    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0ecae0\""
Oct 30 12:24:53.622754 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.622478    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:24:53.660301 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.660259    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8w5\" (UniqueName: \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.660470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.660404    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.783383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.781569    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:53.783383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.781867    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:24:53.783383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.781930    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 12:24:53.784570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.784545    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8w5\" (UniqueName: \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.784816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.784797    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.785906 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.785872    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.819399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.815288    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8w5\" (UniqueName: \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\") pod \"payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:53.923630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:53.923588    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:54.350955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:54.350926    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:24:54.351507 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:24:54.351482    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4fb54c_b6ac_48c3_addd_4378b0524b77.slice/crio-dfd6e5a81f444d0f3cdd88bd023e9bf46377c75882874b45bde1a33ef896f3b6 WatchSource:0}: Error finding container dfd6e5a81f444d0f3cdd88bd023e9bf46377c75882874b45bde1a33ef896f3b6: Status 404 returned error can't find the container with id dfd6e5a81f444d0f3cdd88bd023e9bf46377c75882874b45bde1a33ef896f3b6
Oct 30 12:24:54.770193 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:54.770159    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:24:54.770392 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:54.770213    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 12:24:55.045757 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.045663    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" event={"ID":"3b4fb54c-b6ac-48c3-addd-4378b0524b77","Type":"ContainerStarted","Data":"4f503b519cdc5ac8e82c238664320b22bed7587e2e467f446dc750aa8c08f6d8"}
Oct 30 12:24:55.045931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.045910    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:24:55.045986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.045970    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" event={"ID":"3b4fb54c-b6ac-48c3-addd-4378b0524b77","Type":"ContainerStarted","Data":"dfd6e5a81f444d0f3cdd88bd023e9bf46377c75882874b45bde1a33ef896f3b6"}
Oct 30 12:24:55.084570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.084526    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" podStartSLOduration=2.084512671 podStartE2EDuration="2.084512671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:24:55.080984887 +0000 UTC m=+3294.301269795" watchObservedRunningTime="2026-10-30 12:24:55.084512671 +0000 UTC m=+3294.304797580"
Oct 30 12:24:55.770324 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.770286    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:24:55.770747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:55.770377    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 12:24:56.770424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:56.770381    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:24:56.770835 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:56.770460    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 12:24:57.770388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:57.770353    4453 patch_prober.go:29] interesting pod/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 12:24:57.770576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:57.770408    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 12:24:58.805505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:58.805471    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:58.806965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:58.806941    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:24:58.807842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:24:58.807820    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:25:06.055666 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:06.055631    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:25:07.613716 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:07.613678    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:25:28.065121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.065088    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:28.070889 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.070860    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.099385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.099325    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:28.161672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.161624    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.161882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.161768    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.161882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.161861    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.162027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.161999    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdjj\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.264007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.263963    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.264205 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.264057    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.264205 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.264138    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.264408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.264204    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdjj\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.265051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.265019    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.267041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.267010    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.268115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.268093    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.340762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.296434    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdjj\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\") pod \"payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") " pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.393656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.393621    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:28.901181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:28.892847    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:29.088322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.088221    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:29.101389 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.096839    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.110993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.110961    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:29.213655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.213622    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.213808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.213775    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.213922 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.213904    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6sp\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.213989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.213972    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.315959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.315922    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.316147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.316067    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6sp\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.316221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.316156    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.316320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.316301    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.316870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.316833    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.316970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.316950    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.319199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.319172    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.374781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.374448    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6sp\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\") pod \"payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") " pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.417416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.417375    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:29.445941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.445806    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" event={"ID":"d914d99c-0007-4115-97bc-4e81b0857b99","Type":"ContainerStarted","Data":"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"}
Oct 30 12:25:29.445941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.445897    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" event={"ID":"d914d99c-0007-4115-97bc-4e81b0857b99","Type":"ContainerStarted","Data":"a261b86e9cb9f794b4a866d80d579be9c7752f0b907f2d2186953ddc4bd9b17f"}
Oct 30 12:25:29.445941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.445932    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:29.509057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.508655    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" podStartSLOduration=1.508636091 podStartE2EDuration="1.508636091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:29.499040859 +0000 UTC m=+3328.719325770" watchObservedRunningTime="2026-10-30 12:25:29.508636091 +0000 UTC m=+3328.728921002"
Oct 30 12:25:29.730133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:29.730098    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:29.731918 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:25:29.731892    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c28399e_2817_4d88_8e78_ef4b70277579.slice/crio-bcd15ce2f85bc673fed091989450ece7fef28cdf07d72c40038119f77384dbf0 WatchSource:0}: Error finding container bcd15ce2f85bc673fed091989450ece7fef28cdf07d72c40038119f77384dbf0: Status 404 returned error can't find the container with id bcd15ce2f85bc673fed091989450ece7fef28cdf07d72c40038119f77384dbf0
Oct 30 12:25:30.456672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:30.456638    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" event={"ID":"1c28399e-2817-4d88-8e78-ef4b70277579","Type":"ContainerStarted","Data":"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"}
Oct 30 12:25:30.457133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:30.456698    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" event={"ID":"1c28399e-2817-4d88-8e78-ef4b70277579","Type":"ContainerStarted","Data":"bcd15ce2f85bc673fed091989450ece7fef28cdf07d72c40038119f77384dbf0"}
Oct 30 12:25:30.457133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:30.456934    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:30.496821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:30.496758    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" podStartSLOduration=1.496740513 podStartE2EDuration="1.496740513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:30.492867057 +0000 UTC m=+3329.713151967" watchObservedRunningTime="2026-10-30 12:25:30.496740513 +0000 UTC m=+3329.717025424"
Oct 30 12:25:37.575917 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:37.575880    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:25:39.002090 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.002052    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-59b8a9/agc-cred-5d59545765d0: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:39.002659 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.002638    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:39.002849 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.002822    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials podName:36351911-4cda-4385-8a86-d72116234201 nodeName:}" failed. No retries permitted until 2026-10-30 12:25:39.502800376 +0000 UTC m=+3338.723085284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" (UID: "36351911-4cda-4385-8a86-d72116234201") : secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:39.511178 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.511136    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-59b8a9/agc-cred-5d59545765d0: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:39.511178 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.511166    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:39.511431 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:39.511246    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials podName:36351911-4cda-4385-8a86-d72116234201 nodeName:}" failed. No retries permitted until 2026-10-30 12:25:40.511231645 +0000 UTC m=+3339.731516532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" (UID: "36351911-4cda-4385-8a86-d72116234201") : secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:40.460794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:40.460758    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:40.527647 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:40.527604    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-59b8a9/agc-cred-5d59545765d0: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:40.528155 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:40.528137    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:40.528373 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:40.528357    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials podName:36351911-4cda-4385-8a86-d72116234201 nodeName:}" failed. No retries permitted until 2026-10-30 12:25:42.528317647 +0000 UTC m=+3341.748602557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" (UID: "36351911-4cda-4385-8a86-d72116234201") : secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:40.534977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:40.534954    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:25:41.472214 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:41.472187    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:41.542613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:41.542563    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:25:42.559535 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:42.559500    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-59b8a9/agc-cred-5d59545765d0: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:42.559535 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:42.559526    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw: secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:42.559985 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:42.559588    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials podName:36351911-4cda-4385-8a86-d72116234201 nodeName:}" failed. No retries permitted until 2026-10-30 12:25:46.559573138 +0000 UTC m=+3345.779858026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" (UID: "36351911-4cda-4385-8a86-d72116234201") : secret "agc-cred-5d59545765d0" not found
Oct 30 12:25:45.536911 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.536869    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" podUID="36351911-4cda-4385-8a86-d72116234201" containerName="payload-processing" containerID="cri-o://a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0" gracePeriod=25
Oct 30 12:25:45.825248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.825221    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:25:45.915086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915051    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\") pod \"36351911-4cda-4385-8a86-d72116234201\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") "
Oct 30 12:25:45.915266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915183    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\") pod \"36351911-4cda-4385-8a86-d72116234201\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") "
Oct 30 12:25:45.915266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915241    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\") pod \"36351911-4cda-4385-8a86-d72116234201\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") "
Oct 30 12:25:45.915374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915314    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\") pod \"36351911-4cda-4385-8a86-d72116234201\" (UID: \"36351911-4cda-4385-8a86-d72116234201\") "
Oct 30 12:25:45.915456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915431    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume" pod "36351911-4cda-4385-8a86-d72116234201" (UID: "36351911-4cda-4385-8a86-d72116234201"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:45.915693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.915671    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume" pod "36351911-4cda-4385-8a86-d72116234201" (UID: "36351911-4cda-4385-8a86-d72116234201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:45.916165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.916136    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:45.916280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.916165    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36351911-4cda-4385-8a86-d72116234201-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:45.917454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.917434    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw" pod "36351911-4cda-4385-8a86-d72116234201" (UID: "36351911-4cda-4385-8a86-d72116234201"). InnerVolumeSpecName "kube-api-access-rtnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:45.917520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:45.917465    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials" pod "36351911-4cda-4385-8a86-d72116234201" (UID: "36351911-4cda-4385-8a86-d72116234201"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:46.017511 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.017472    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtnvw\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-kube-api-access-rtnvw\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:46.017511 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.017503    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36351911-4cda-4385-8a86-d72116234201-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:46.542952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.542914    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" podUID="1f225d10-33d5-459f-907a-020e3c053bd3" containerName="payload-processing" containerID="cri-o://2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a" gracePeriod=25
Oct 30 12:25:46.612181 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.612151    4453 generic.go:356] "Generic (PLEG): container finished" podID="36351911-4cda-4385-8a86-d72116234201" containerID="a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0" exitCode=0
Oct 30 12:25:46.612322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.612197    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"
Oct 30 12:25:46.612322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.612226    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" event={"ID":"36351911-4cda-4385-8a86-d72116234201","Type":"ContainerDied","Data":"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"}
Oct 30 12:25:46.612322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.612280    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw" event={"ID":"36351911-4cda-4385-8a86-d72116234201","Type":"ContainerDied","Data":"26e86d5a6edc624a8f3b8591a6153a4ebf66ff5cbfc5699d8ef07a159a59628e"}
Oct 30 12:25:46.612322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.612303    4453 scope.go:122] "RemoveContainer" containerID="a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"
Oct 30 12:25:46.656673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.656652    4453 scope.go:122] "RemoveContainer" containerID="a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"
Oct 30 12:25:46.656991 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:46.656967    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0\": container with ID starting with a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0 not found: ID does not exist" containerID="a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"
Oct 30 12:25:46.657077 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.656997    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0"} err="failed to get container status \"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0\": rpc error: code = NotFound desc = could not find container \"a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0\": container with ID starting with a0636e2bb30f3e931d2c136583900bb20c66915dbfcd52402958a39053fe8cd0 not found: ID does not exist"
Oct 30 12:25:46.685520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.685492    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:25:46.701101 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.701067    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-8467l5ksw"]
Oct 30 12:25:46.846526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.846500    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:25:46.917946 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.917908    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36351911-4cda-4385-8a86-d72116234201" path="/var/lib/kubelet/pods/36351911-4cda-4385-8a86-d72116234201/volumes"
Oct 30 12:25:46.930282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.930250    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\") pod \"1f225d10-33d5-459f-907a-020e3c053bd3\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") "
Oct 30 12:25:46.930532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.930504    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\") pod \"1f225d10-33d5-459f-907a-020e3c053bd3\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") "
Oct 30 12:25:46.930675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.930659    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\") pod \"1f225d10-33d5-459f-907a-020e3c053bd3\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") "
Oct 30 12:25:46.930800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.930783    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\") pod \"1f225d10-33d5-459f-907a-020e3c053bd3\" (UID: \"1f225d10-33d5-459f-907a-020e3c053bd3\") "
Oct 30 12:25:46.931768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.931744    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume" pod "1f225d10-33d5-459f-907a-020e3c053bd3" (UID: "1f225d10-33d5-459f-907a-020e3c053bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:46.932496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.932469    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume" pod "1f225d10-33d5-459f-907a-020e3c053bd3" (UID: "1f225d10-33d5-459f-907a-020e3c053bd3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:46.935484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.935457    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials" pod "1f225d10-33d5-459f-907a-020e3c053bd3" (UID: "1f225d10-33d5-459f-907a-020e3c053bd3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:46.935601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:46.935499    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk" pod "1f225d10-33d5-459f-907a-020e3c053bd3" (UID: "1f225d10-33d5-459f-907a-020e3c053bd3"). InnerVolumeSpecName "kube-api-access-bbstk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:47.033111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.033082    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbstk\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-kube-api-access-bbstk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:47.033111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.033110    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:47.033303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.033122    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f225d10-33d5-459f-907a-020e3c053bd3-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:47.033303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.033135    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f225d10-33d5-459f-907a-020e3c053bd3-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:47.622411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.622373    4453 generic.go:356] "Generic (PLEG): container finished" podID="1f225d10-33d5-459f-907a-020e3c053bd3" containerID="2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a" exitCode=0
Oct 30 12:25:47.622842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.622418    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"
Oct 30 12:25:47.622842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.622457    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" event={"ID":"1f225d10-33d5-459f-907a-020e3c053bd3","Type":"ContainerDied","Data":"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"}
Oct 30 12:25:47.622842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.622517    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv" event={"ID":"1f225d10-33d5-459f-907a-020e3c053bd3","Type":"ContainerDied","Data":"322215ccd2b87bb48af8fa83bfce83974e9af8c79221d43638e3378df7acb34d"}
Oct 30 12:25:47.622842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.622548    4453 scope.go:122] "RemoveContainer" containerID="2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"
Oct 30 12:25:47.633852 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.633834    4453 scope.go:122] "RemoveContainer" containerID="2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"
Oct 30 12:25:47.634161 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:47.634138    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a\": container with ID starting with 2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a not found: ID does not exist" containerID="2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"
Oct 30 12:25:47.634261 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.634165    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a"} err="failed to get container status \"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a\": rpc error: code = NotFound desc = could not find container \"2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a\": container with ID starting with 2d5a5071aad8543725e7c016fc8b9f26d515070a3b70b46b7fd6c28f0aeb420a not found: ID does not exist"
Oct 30 12:25:47.669051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.669025    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:25:47.678197 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:47.678170    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-6f966h9tv"]
Oct 30 12:25:48.914012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:48.913979    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f225d10-33d5-459f-907a-020e3c053bd3" path="/var/lib/kubelet/pods/1f225d10-33d5-459f-907a-020e3c053bd3/volumes"
Oct 30 12:25:50.672454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:50.672419    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:51.618481 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:51.618446    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:55.432070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.432042    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:25:55.433923 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.433893    4453 cpu_manager.go:414] "containerMap: removing container" podUID="36351911-4cda-4385-8a86-d72116234201" containerName="payload-processing"
Oct 30 12:25:55.434119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.434102    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36351911-4cda-4385-8a86-d72116234201" containerName="payload-processing"
Oct 30 12:25:55.434249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.434236    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1f225d10-33d5-459f-907a-020e3c053bd3" containerName="payload-processing"
Oct 30 12:25:55.434354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.434323    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f225d10-33d5-459f-907a-020e3c053bd3" containerName="payload-processing"
Oct 30 12:25:55.434635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.434622    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="36351911-4cda-4385-8a86-d72116234201" containerName="payload-processing"
Oct 30 12:25:55.434726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.434716    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f225d10-33d5-459f-907a-020e3c053bd3" containerName="payload-processing"
Oct 30 12:25:55.440459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.440439    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:25:55.440612 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.440589    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.521952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.521909    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.522147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.522097    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzjk\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.522253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.522235    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.522372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.522345    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.626355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.625589    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.626355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.625673    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.626355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.625837    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.626355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.625970    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzjk\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.631349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.627322    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.642030 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.628320    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.661293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.661261    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzjk\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.665379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.665308    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.673471 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.673428    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" podUID="d914d99c-0007-4115-97bc-4e81b0857b99" containerName="payload-processing" containerID="cri-o://c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d" gracePeriod=25
Oct 30 12:25:55.765879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.765834    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:55.984914 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:55.984888    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:56.038081 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.038041    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\") pod \"d914d99c-0007-4115-97bc-4e81b0857b99\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") "
Oct 30 12:25:56.038316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.038099    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\") pod \"d914d99c-0007-4115-97bc-4e81b0857b99\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") "
Oct 30 12:25:56.038316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.038138    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\") pod \"d914d99c-0007-4115-97bc-4e81b0857b99\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") "
Oct 30 12:25:56.038316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.038191    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\") pod \"d914d99c-0007-4115-97bc-4e81b0857b99\" (UID: \"d914d99c-0007-4115-97bc-4e81b0857b99\") "
Oct 30 12:25:56.039266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.039228    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume" pod "d914d99c-0007-4115-97bc-4e81b0857b99" (UID: "d914d99c-0007-4115-97bc-4e81b0857b99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:56.039424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.039239    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume" pod "d914d99c-0007-4115-97bc-4e81b0857b99" (UID: "d914d99c-0007-4115-97bc-4e81b0857b99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:56.041053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.041026    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials" pod "d914d99c-0007-4115-97bc-4e81b0857b99" (UID: "d914d99c-0007-4115-97bc-4e81b0857b99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:56.041561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.041531    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj" pod "d914d99c-0007-4115-97bc-4e81b0857b99" (UID: "d914d99c-0007-4115-97bc-4e81b0857b99"). InnerVolumeSpecName "kube-api-access-9cdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:56.081598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.081566    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:25:56.082039 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:25:56.082015    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3e93d0_3a9c_4823_b445_f1967d0244a2.slice/crio-f35b18c61ecb57ff336aec47ff8a9b2134f3baaa0c72884b89a15d2f8a78f55b WatchSource:0}: Error finding container f35b18c61ecb57ff336aec47ff8a9b2134f3baaa0c72884b89a15d2f8a78f55b: Status 404 returned error can't find the container with id f35b18c61ecb57ff336aec47ff8a9b2134f3baaa0c72884b89a15d2f8a78f55b
Oct 30 12:25:56.140557 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.140527    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.140557 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.140554    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.140557 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.140565    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d914d99c-0007-4115-97bc-4e81b0857b99-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.140872 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.140574    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdjj\" (UniqueName: \"kubernetes.io/projected/d914d99c-0007-4115-97bc-4e81b0857b99-kube-api-access-9cdjj\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.178164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.178131    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:25:56.179106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.179086    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d914d99c-0007-4115-97bc-4e81b0857b99" containerName="payload-processing"
Oct 30 12:25:56.179106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.179106    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d914d99c-0007-4115-97bc-4e81b0857b99" containerName="payload-processing"
Oct 30 12:25:56.179258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.179248    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d914d99c-0007-4115-97bc-4e81b0857b99" containerName="payload-processing"
Oct 30 12:25:56.182970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.182946    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.204248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.204217    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:25:56.241696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.241597    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcn8v\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.241871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.241835    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.241979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.241956    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.242155 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.242046    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.343645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.343601    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.343850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.343685    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.343850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.343735    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.343979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.343906    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcn8v\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.344563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.344541    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.344721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.344697    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.346124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.346100    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.364254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.364229    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcn8v\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.500574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.500496    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:56.619465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.619413    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" podUID="1c28399e-2817-4d88-8e78-ef4b70277579" containerName="payload-processing" containerID="cri-o://0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806" gracePeriod=25
Oct 30 12:25:56.731956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.731584    4453 generic.go:356] "Generic (PLEG): container finished" podID="d914d99c-0007-4115-97bc-4e81b0857b99" containerID="c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d" exitCode=0
Oct 30 12:25:56.731956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.731643    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"
Oct 30 12:25:56.731956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.731677    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" event={"ID":"d914d99c-0007-4115-97bc-4e81b0857b99","Type":"ContainerDied","Data":"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"}
Oct 30 12:25:56.731956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.731735    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf" event={"ID":"d914d99c-0007-4115-97bc-4e81b0857b99","Type":"ContainerDied","Data":"a261b86e9cb9f794b4a866d80d579be9c7752f0b907f2d2186953ddc4bd9b17f"}
Oct 30 12:25:56.731956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.731760    4453 scope.go:122] "RemoveContainer" containerID="c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"
Oct 30 12:25:56.733918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.733885    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" event={"ID":"4e3e93d0-3a9c-4823-b445-f1967d0244a2","Type":"ContainerStarted","Data":"65222af90981c3c5293e4450c01d100c98e9de66383829a55e853c57e632e6af"}
Oct 30 12:25:56.734075 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.733949    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:25:56.734075 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.733976    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" event={"ID":"4e3e93d0-3a9c-4823-b445-f1967d0244a2","Type":"ContainerStarted","Data":"f35b18c61ecb57ff336aec47ff8a9b2134f3baaa0c72884b89a15d2f8a78f55b"}
Oct 30 12:25:56.742478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.742458    4453 scope.go:122] "RemoveContainer" containerID="c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"
Oct 30 12:25:56.742740 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:56.742720    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d\": container with ID starting with c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d not found: ID does not exist" containerID="c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"
Oct 30 12:25:56.742792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.742748    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d"} err="failed to get container status \"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d\": rpc error: code = NotFound desc = could not find container \"c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d\": container with ID starting with c360635213a54be05fa84cbe3b3c2763770fcf3153991cd680c165749eced34d not found: ID does not exist"
Oct 30 12:25:56.786326 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.786236    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" podStartSLOduration=1.786221624 podStartE2EDuration="1.786221624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:56.78273451 +0000 UTC m=+3356.003019420" watchObservedRunningTime="2026-10-30 12:25:56.786221624 +0000 UTC m=+3356.006506569"
Oct 30 12:25:56.810044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.810005    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:25:56.816276 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:25:56.816239    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3b9304_c193_4ea1_a5c6_e43e7a3efc4d.slice/crio-f5bf84891e79ddd1ce7e301897a34dde8cae8e63456b68f0d23999fb2c06b548 WatchSource:0}: Error finding container f5bf84891e79ddd1ce7e301897a34dde8cae8e63456b68f0d23999fb2c06b548: Status 404 returned error can't find the container with id f5bf84891e79ddd1ce7e301897a34dde8cae8e63456b68f0d23999fb2c06b548
Oct 30 12:25:56.829602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.829544    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:56.834139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.834116    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-59b8a9/payload-processing-external-model-e2e-extmod-a-59b8a9-7bc9dc6rf"]
Oct 30 12:25:56.914562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.914533    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d914d99c-0007-4115-97bc-4e81b0857b99" path="/var/lib/kubelet/pods/d914d99c-0007-4115-97bc-4e81b0857b99/volumes"
Oct 30 12:25:56.925118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.925090    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:56.969625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.969592    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\") pod \"1c28399e-2817-4d88-8e78-ef4b70277579\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") "
Oct 30 12:25:56.969799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.969712    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\") pod \"1c28399e-2817-4d88-8e78-ef4b70277579\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") "
Oct 30 12:25:56.969998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.969978    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\") pod \"1c28399e-2817-4d88-8e78-ef4b70277579\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") "
Oct 30 12:25:56.970094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.970021    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume" pod "1c28399e-2817-4d88-8e78-ef4b70277579" (UID: "1c28399e-2817-4d88-8e78-ef4b70277579"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:56.970168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.970121    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\") pod \"1c28399e-2817-4d88-8e78-ef4b70277579\" (UID: \"1c28399e-2817-4d88-8e78-ef4b70277579\") "
Oct 30 12:25:56.970961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.970705    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume" pod "1c28399e-2817-4d88-8e78-ef4b70277579" (UID: "1c28399e-2817-4d88-8e78-ef4b70277579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:25:56.970961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.970820    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.970961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.970845    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c28399e-2817-4d88-8e78-ef4b70277579-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:56.972291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.972266    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp" pod "1c28399e-2817-4d88-8e78-ef4b70277579" (UID: "1c28399e-2817-4d88-8e78-ef4b70277579"). InnerVolumeSpecName "kube-api-access-hk6sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:56.972494 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:56.972473    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials" pod "1c28399e-2817-4d88-8e78-ef4b70277579" (UID: "1c28399e-2817-4d88-8e78-ef4b70277579"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:25:57.072076 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.072004    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:57.072076 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.072032    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6sp\" (UniqueName: \"kubernetes.io/projected/1c28399e-2817-4d88-8e78-ef4b70277579-kube-api-access-hk6sp\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:25:57.744092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.744054    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" event={"ID":"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d","Type":"ContainerStarted","Data":"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"}
Oct 30 12:25:57.744562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.744119    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:25:57.744562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.744141    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" event={"ID":"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d","Type":"ContainerStarted","Data":"f5bf84891e79ddd1ce7e301897a34dde8cae8e63456b68f0d23999fb2c06b548"}
Oct 30 12:25:57.747264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.747228    4453 generic.go:356] "Generic (PLEG): container finished" podID="1c28399e-2817-4d88-8e78-ef4b70277579" containerID="0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806" exitCode=0
Oct 30 12:25:57.747453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.747295    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"
Oct 30 12:25:57.747453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.747325    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" event={"ID":"1c28399e-2817-4d88-8e78-ef4b70277579","Type":"ContainerDied","Data":"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"}
Oct 30 12:25:57.747453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.747420    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28" event={"ID":"1c28399e-2817-4d88-8e78-ef4b70277579","Type":"ContainerDied","Data":"bcd15ce2f85bc673fed091989450ece7fef28cdf07d72c40038119f77384dbf0"}
Oct 30 12:25:57.747453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.747450    4453 scope.go:122] "RemoveContainer" containerID="0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"
Oct 30 12:25:57.758841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.758814    4453 scope.go:122] "RemoveContainer" containerID="0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"
Oct 30 12:25:57.759095 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:25:57.759076    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806\": container with ID starting with 0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806 not found: ID does not exist" containerID="0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"
Oct 30 12:25:57.759311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.759102    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806"} err="failed to get container status \"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806\": rpc error: code = NotFound desc = could not find container \"0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806\": container with ID starting with 0bb2118d1e3db56c6b752ecb9fb4a29c144a690d1dac4180cb088c5e53faa806 not found: ID does not exist"
Oct 30 12:25:57.781763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.781650    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" podStartSLOduration=1.781634032 podStartE2EDuration="1.781634032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:25:57.780218479 +0000 UTC m=+3357.000503390" watchObservedRunningTime="2026-10-30 12:25:57.781634032 +0000 UTC m=+3357.001918945"
Oct 30 12:25:57.813240 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.813215    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:57.827146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:57.827116    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b15ec3/payload-processing-external-model-e2e-extmod-b-b15ec3-5787t5x28"]
Oct 30 12:25:59.038114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:25:59.038072    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c28399e-2817-4d88-8e78-ef4b70277579" path="/var/lib/kubelet/pods/1c28399e-2817-4d88-8e78-ef4b70277579/volumes"
Oct 30 12:26:04.565021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.564983    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:26:04.565798 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.565783    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1c28399e-2817-4d88-8e78-ef4b70277579" containerName="payload-processing"
Oct 30 12:26:04.565798 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.565799    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c28399e-2817-4d88-8e78-ef4b70277579" containerName="payload-processing"
Oct 30 12:26:04.565953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.565942    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c28399e-2817-4d88-8e78-ef4b70277579" containerName="payload-processing"
Oct 30 12:26:04.571510 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.571491    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.589991 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.589968    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:26:04.595291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.595265    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0190d8\""
Oct 30 12:26:04.595519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.595498    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0190d8-dockercfg-92ljs\""
Oct 30 12:26:04.718356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.717171    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:26:04.744354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.743628    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjb6c\" (UniqueName: \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.758356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.753430    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.788353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.785562    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:26:04.788353 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.785730    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:04.892171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.888570    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:04.892171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.888706    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.892171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.888788    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rh49\" (UniqueName: \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:04.892171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.889069    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjb6c\" (UniqueName: \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.892171 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.891988    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.912935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.912904    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjb6c\" (UniqueName: \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\") pod \"payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:04.990920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.990887    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:04.991098 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.990982    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rh49\" (UniqueName: \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:04.991749 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:04.991723    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:05.012618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.012586    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rh49\" (UniqueName: \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\") pod \"payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:05.186579 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.186506    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:05.214705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.213589    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:05.494969 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.494934    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:26:05.496995 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:26:05.496969    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8443e82_cb7c_4042_a03f_0e51d8160358.slice/crio-6cd44dcb050c4099f6973a7718edc17fd7332f7ca01c07a4925cbb82f1fe537c WatchSource:0}: Error finding container 6cd44dcb050c4099f6973a7718edc17fd7332f7ca01c07a4925cbb82f1fe537c: Status 404 returned error can't find the container with id 6cd44dcb050c4099f6973a7718edc17fd7332f7ca01c07a4925cbb82f1fe537c
Oct 30 12:26:05.544035 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:26:05.544009    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828dc1ab_5579_4734_b6ae_3cd75432c9dd.slice/crio-4fef4215b6894f5cb0c68a94affb52befd0e980db1044dac113e10d88adbc148 WatchSource:0}: Error finding container 4fef4215b6894f5cb0c68a94affb52befd0e980db1044dac113e10d88adbc148: Status 404 returned error can't find the container with id 4fef4215b6894f5cb0c68a94affb52befd0e980db1044dac113e10d88adbc148
Oct 30 12:26:05.544429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.544405    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:26:05.903963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.903930    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" event={"ID":"f8443e82-cb7c-4042-a03f-0e51d8160358","Type":"ContainerStarted","Data":"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"}
Oct 30 12:26:05.904478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.903977    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" event={"ID":"f8443e82-cb7c-4042-a03f-0e51d8160358","Type":"ContainerStarted","Data":"6cd44dcb050c4099f6973a7718edc17fd7332f7ca01c07a4925cbb82f1fe537c"}
Oct 30 12:26:05.905713 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.905685    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" event={"ID":"828dc1ab-5579-4734-b6ae-3cd75432c9dd","Type":"ContainerStarted","Data":"c5cd0dde83d28018cd19e5f21de1c793fcd9255ff904e710122591394881ae93"}
Oct 30 12:26:05.905828 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.905728    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" event={"ID":"828dc1ab-5579-4734-b6ae-3cd75432c9dd","Type":"ContainerStarted","Data":"4fef4215b6894f5cb0c68a94affb52befd0e980db1044dac113e10d88adbc148"}
Oct 30 12:26:05.905828 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.905757    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:05.939178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.939133    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" podStartSLOduration=1.939119835 podStartE2EDuration="1.939119835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:05.938681239 +0000 UTC m=+3365.158966153" watchObservedRunningTime="2026-10-30 12:26:05.939119835 +0000 UTC m=+3365.159404745"
Oct 30 12:26:05.976655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:05.976592    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" podStartSLOduration=1.976574273 podStartE2EDuration="1.976574273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:05.973214552 +0000 UTC m=+3365.193499462" watchObservedRunningTime="2026-10-30 12:26:05.976574273 +0000 UTC m=+3365.196859185"
Oct 30 12:26:06.914934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:06.914897    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:07.571087 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:07.571053    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:26:07.750763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:07.750734    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:26:07.827189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:07.827087    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:26:08.762148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:08.762098    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:26:08.886354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:08.873890    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:26:12.828037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:12.827974    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" podUID="369c5a51-3926-4d61-8d51-26194c9f8988" containerName="payload-processing" containerID="cri-o://ddce7fd1bedb4d183a7e69272f201d7b5beeef73c5159620fa6b337637224001" gracePeriod=25
Oct 30 12:26:13.020776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.020738    4453 generic.go:356] "Generic (PLEG): container finished" podID="369c5a51-3926-4d61-8d51-26194c9f8988" containerID="ddce7fd1bedb4d183a7e69272f201d7b5beeef73c5159620fa6b337637224001" exitCode=0
Oct 30 12:26:13.021006 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.020985    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" event={"ID":"369c5a51-3926-4d61-8d51-26194c9f8988","Type":"ContainerDied","Data":"ddce7fd1bedb4d183a7e69272f201d7b5beeef73c5159620fa6b337637224001"}
Oct 30 12:26:13.145592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.145564    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:26:13.247904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.247873    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\") pod \"369c5a51-3926-4d61-8d51-26194c9f8988\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") "
Oct 30 12:26:13.248132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.247993    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\") pod \"369c5a51-3926-4d61-8d51-26194c9f8988\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") "
Oct 30 12:26:13.248132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.248042    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\") pod \"369c5a51-3926-4d61-8d51-26194c9f8988\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") "
Oct 30 12:26:13.248132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.248122    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\") pod \"369c5a51-3926-4d61-8d51-26194c9f8988\" (UID: \"369c5a51-3926-4d61-8d51-26194c9f8988\") "
Oct 30 12:26:13.248560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.248536    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume" pod "369c5a51-3926-4d61-8d51-26194c9f8988" (UID: "369c5a51-3926-4d61-8d51-26194c9f8988"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:13.248560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.248547    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume" pod "369c5a51-3926-4d61-8d51-26194c9f8988" (UID: "369c5a51-3926-4d61-8d51-26194c9f8988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:13.249055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.249032    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:13.249155 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.249057    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/369c5a51-3926-4d61-8d51-26194c9f8988-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:13.251198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.251176    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz" pod "369c5a51-3926-4d61-8d51-26194c9f8988" (UID: "369c5a51-3926-4d61-8d51-26194c9f8988"). InnerVolumeSpecName "kube-api-access-fqlrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:13.251310 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.251207    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials" pod "369c5a51-3926-4d61-8d51-26194c9f8988" (UID: "369c5a51-3926-4d61-8d51-26194c9f8988"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:13.352453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.350752    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:13.352453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.350793    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlrz\" (UniqueName: \"kubernetes.io/projected/369c5a51-3926-4d61-8d51-26194c9f8988-kube-api-access-fqlrz\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:13.875072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:13.874987    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" podUID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" containerName="payload-processing" containerID="cri-o://37e74ef0ffa6c2bdd173d4a88bf2291a7913af63c146e47c6976d0065febfaba" gracePeriod=25
Oct 30 12:26:14.030692 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.030661    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"
Oct 30 12:26:14.030692 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.030671    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz" event={"ID":"369c5a51-3926-4d61-8d51-26194c9f8988","Type":"ContainerDied","Data":"119f18d4eaafb8e80a308d2d12d2ffbeae06dfa6b93f71b6af15c736877adc60"}
Oct 30 12:26:14.030935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.030741    4453 scope.go:122] "RemoveContainer" containerID="ddce7fd1bedb4d183a7e69272f201d7b5beeef73c5159620fa6b337637224001"
Oct 30 12:26:14.032613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.032592    4453 generic.go:356] "Generic (PLEG): container finished" podID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" containerID="37e74ef0ffa6c2bdd173d4a88bf2291a7913af63c146e47c6976d0065febfaba" exitCode=0
Oct 30 12:26:14.032741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.032669    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" event={"ID":"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b","Type":"ContainerDied","Data":"37e74ef0ffa6c2bdd173d4a88bf2291a7913af63c146e47c6976d0065febfaba"}
Oct 30 12:26:14.077945 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.077907    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:26:14.088104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.088077    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-6658mdmfz"]
Oct 30 12:26:14.272529 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.272503    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:26:14.367323 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.367285    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\") pod \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") "
Oct 30 12:26:14.367527 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.367444    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\") pod \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") "
Oct 30 12:26:14.367594 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.367556    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\") pod \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") "
Oct 30 12:26:14.367594 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.367587    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\") pod \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\" (UID: \"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b\") "
Oct 30 12:26:14.368136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.368105    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume" pod "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" (UID: "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:14.368278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.368222    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:14.368278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.368260    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume" pod "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" (UID: "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:14.369661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.369638    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials" pod "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" (UID: "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:14.369769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.369672    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb" pod "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" (UID: "a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b"). InnerVolumeSpecName "kube-api-access-949jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:14.469429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.469392    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:14.469429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.469422    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949jb\" (UniqueName: \"kubernetes.io/projected/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-kube-api-access-949jb\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:14.469429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.469431    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:14.923470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:14.923434    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369c5a51-3926-4d61-8d51-26194c9f8988" path="/var/lib/kubelet/pods/369c5a51-3926-4d61-8d51-26194c9f8988/volumes"
Oct 30 12:26:15.042796 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:15.042767    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"
Oct 30 12:26:15.042796 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:15.042786    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws" event={"ID":"a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b","Type":"ContainerDied","Data":"01e46c55d11ba8d71f14ae003ac83294eaa8f174a49cb476e30dc00f1486e644"}
Oct 30 12:26:15.043083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:15.042841    4453 scope.go:122] "RemoveContainer" containerID="37e74ef0ffa6c2bdd173d4a88bf2291a7913af63c146e47c6976d0065febfaba"
Oct 30 12:26:15.122354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:15.119380    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:26:15.142356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:15.137738    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-7c8f9wrws"]
Oct 30 12:26:16.936832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:16.934842    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" path="/var/lib/kubelet/pods/a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b/volumes"
Oct 30 12:26:16.936832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:16.935250    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:26:17.924164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:17.924130    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:26:18.423455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.423414    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:18.424491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424458    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" containerName="payload-processing"
Oct 30 12:26:18.424491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424483    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" containerName="payload-processing"
Oct 30 12:26:18.424491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424495    4453 cpu_manager.go:414] "containerMap: removing container" podUID="369c5a51-3926-4d61-8d51-26194c9f8988" containerName="payload-processing"
Oct 30 12:26:18.424752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424500    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369c5a51-3926-4d61-8d51-26194c9f8988" containerName="payload-processing"
Oct 30 12:26:18.424752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424669    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a7b4f7-f700-4972-9f32-b1b5ecf32f4b" containerName="payload-processing"
Oct 30 12:26:18.424752 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.424680    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="369c5a51-3926-4d61-8d51-26194c9f8988" containerName="payload-processing"
Oct 30 12:26:18.483621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.483580    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:18.483803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.483716    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.504961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.504933    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.505109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.505002    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.505246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.505224    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74kbz\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.505309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.505274    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.606880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.606847    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74kbz\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.606880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.606887    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.607073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.607029    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.607121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.607081    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.607690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.607667    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.607802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.607704    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.609564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.609543    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.630294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.630267    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74kbz\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\") pod \"payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") " pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:18.896055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:18.896014    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:19.292489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.292078    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:19.355031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.355002    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"]
Oct 30 12:26:19.494426 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.494390    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"]
Oct 30 12:26:19.494825 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.494499    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.584326 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.584281    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.584517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.584423    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwrl\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.584598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.584521    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.584779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.584756    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.686675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.686638    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.686894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.686693    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwrl\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.686894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.686772    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.687024 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:19.686901    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-5d74e357e0f1: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:19.687024 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:19.686923    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:19.687024 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.686961    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.687024 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:19.686985    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials podName:06e84aa4-0283-4c30-ad47-ac068b287f28 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:20.186963344 +0000 UTC m=+3379.407248236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28") : secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:19.687574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.687554    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.687626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.687557    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.708088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.708059    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwrl\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:19.901358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.894240    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:26:19.908037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:19.903255    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:20.098228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.098194    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" event={"ID":"0e887d3e-604b-4ed0-ad09-ad520c9f71c8","Type":"ContainerStarted","Data":"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"}
Oct 30 12:26:20.098512 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.098255    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" event={"ID":"0e887d3e-604b-4ed0-ad09-ad520c9f71c8","Type":"ContainerStarted","Data":"b1cf39509a3097d9397d01dd0111e428ba673d74a88e7db4bc457de62f158a52"}
Oct 30 12:26:20.098512 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.098420    4453 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" secret="" err="secret \"payload-processing-external-model-e2e--793d1056-dockercfg-zj5q5\" not found"
Oct 30 12:26:20.098512 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.098432    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:20.135104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.135042    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" podStartSLOduration=2.135026572 podStartE2EDuration="2.135026572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:20.132979657 +0000 UTC m=+3379.353264566" watchObservedRunningTime="2026-10-30 12:26:20.135026572 +0000 UTC m=+3379.355311488"
Oct 30 12:26:20.198758 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.198666    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:20.198931 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198759    4453 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:20.198931 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198832    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:20.698813682 +0000 UTC m=+3379.919098592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:20.198931 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198875    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bc4e2/agc-cred-972432cc9b49: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.198931 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198888    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.198931 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198925    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:20.698913335 +0000 UTC m=+3379.919198243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.199263 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198977    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-5d74e357e0f1: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:20.199263 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.198987    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:20.199263 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.199016    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials podName:06e84aa4-0283-4c30-ad47-ac068b287f28 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:21.199006156 +0000 UTC m=+3380.419291064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28") : secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:20.744261 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.744222    4453 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:20.744776 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.744301    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:21.744284874 +0000 UTC m=+3380.964569785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:20.744776 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.744416    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bc4e2/agc-cred-972432cc9b49: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.744776 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.744434    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.744776 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.744484    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:21.744469141 +0000 UTC m=+3380.964754030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:20.760061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.760002    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:26:20.771798 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.771757    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"]
Oct 30 12:26:20.772480 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:20.772449    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb" podUID="06e84aa4-0283-4c30-ad47-ac068b287f28"
Oct 30 12:26:20.983481 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.983443    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:26:20.983661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:20.983497    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:26:21.105953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.105869    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:21.183762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.183723    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:21.256416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.253822    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\") pod \"06e84aa4-0283-4c30-ad47-ac068b287f28\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") "
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.256728    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\") pod \"06e84aa4-0283-4c30-ad47-ac068b287f28\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") "
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.256931    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\") pod \"06e84aa4-0283-4c30-ad47-ac068b287f28\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") "
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.257364    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb\" (UID: \"06e84aa4-0283-4c30-ad47-ac068b287f28\") " pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.257932    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-f8f132/agc-cred-5d74e357e0f1: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.257951    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb: secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.258024    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials podName:06e84aa4-0283-4c30-ad47-ac068b287f28 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:23.2580054 +0000 UTC m=+3382.478290290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28") : secret "agc-cred-5d74e357e0f1" not found
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.266470    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume" pod "06e84aa4-0283-4c30-ad47-ac068b287f28" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:21.277377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.266853    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume" pod "06e84aa4-0283-4c30-ad47-ac068b287f28" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:21.332418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.324240    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl" pod "06e84aa4-0283-4c30-ad47-ac068b287f28" (UID: "06e84aa4-0283-4c30-ad47-ac068b287f28"). InnerVolumeSpecName "kube-api-access-rxwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:21.368297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.367435    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxwrl\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-kube-api-access-rxwrl\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:21.368297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.367470    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:21.368297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:21.367486    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06e84aa4-0283-4c30-ad47-ac068b287f28-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:21.774002 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.773971    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bc4e2/agc-cred-972432cc9b49: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:21.774002 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.774002    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:21.774469 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.774046    4453 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:21.774469 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.774074    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:23.774051501 +0000 UTC m=+3382.994336390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:21.774469 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:21.774093    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:23.774083447 +0000 UTC m=+3382.994368341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:22.127189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.123857    4453 generic.go:356] "Generic (PLEG): container finished" podID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerID="08d2cd3d089fd401e7dd0b88d204f51aa65945d80704d8d893c04ef288e58846" exitCode=0
Oct 30 12:26:22.127189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.124285    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"
Oct 30 12:26:22.127189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.125622    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerDied","Data":"08d2cd3d089fd401e7dd0b88d204f51aa65945d80704d8d893c04ef288e58846"}
Oct 30 12:26:22.127189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.126582    4453 scope.go:122] "RemoveContainer" containerID="08d2cd3d089fd401e7dd0b88d204f51aa65945d80704d8d893c04ef288e58846"
Oct 30 12:26:22.129112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.129070    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:22.129428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.129163    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:22.129428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.129214    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:22.234636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.234605    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"]
Oct 30 12:26:22.241644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.241612    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-b96cj9sxb"]
Oct 30 12:26:22.284137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.284097    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06e84aa4-0283-4c30-ad47-ac068b287f28-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:22.913672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:22.913642    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e84aa4-0283-4c30-ad47-ac068b287f28" path="/var/lib/kubelet/pods/06e84aa4-0283-4c30-ad47-ac068b287f28/volumes"
Oct 30 12:26:23.135135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:23.135094    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerStarted","Data":"618fbc7de1663990e7b5bd1f63fb10e274a4138369c79f430f9f1c60986f7487"}
Oct 30 12:26:23.565683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:23.565652    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:26:23.565855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:23.565770    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:23.565855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:23.565816    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:23.819850 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:23.819765    4453 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-2bc4e2/agc-cred-972432cc9b49: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:23.819850 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:23.819789    4453 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c: secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:23.819850 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:23.819841    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:27.819823915 +0000 UTC m=+3387.040108808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : secret "agc-cred-972432cc9b49" not found
Oct 30 12:26:23.820121 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:23.819875    4453 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2: configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:23.820121 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:23.819975    4453 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume podName:0e887d3e-604b-4ed0-ad09-ad520c9f71c8 nodeName:}" failed. No retries permitted until 2026-10-30 12:26:27.81995775 +0000 UTC m=+3387.040242654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-2bc4e2" not found
Oct 30 12:26:24.566018 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:24.565988    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:24.566414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:24.566036    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:24.901360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:24.898389    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" podUID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" containerName="payload-processing" containerID="cri-o://65222af90981c3c5293e4450c01d100c98e9de66383829a55e853c57e632e6af" gracePeriod=25
Oct 30 12:26:25.155235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.155080    4453 generic.go:356] "Generic (PLEG): container finished" podID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" containerID="65222af90981c3c5293e4450c01d100c98e9de66383829a55e853c57e632e6af" exitCode=0
Oct 30 12:26:25.155396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.155237    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" event={"ID":"4e3e93d0-3a9c-4823-b445-f1967d0244a2","Type":"ContainerDied","Data":"65222af90981c3c5293e4450c01d100c98e9de66383829a55e853c57e632e6af"}
Oct 30 12:26:25.266294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.266262    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:26:25.352777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.352741    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\") pod \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") "
Oct 30 12:26:25.352979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.352909    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\") pod \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") "
Oct 30 12:26:25.353060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.352988    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\") pod \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") "
Oct 30 12:26:25.353060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.353042    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\") pod \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\" (UID: \"4e3e93d0-3a9c-4823-b445-f1967d0244a2\") "
Oct 30 12:26:25.353247 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.353215    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume" pod "4e3e93d0-3a9c-4823-b445-f1967d0244a2" (UID: "4e3e93d0-3a9c-4823-b445-f1967d0244a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:25.353709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.353678    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume" pod "4e3e93d0-3a9c-4823-b445-f1967d0244a2" (UID: "4e3e93d0-3a9c-4823-b445-f1967d0244a2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:25.354372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.354300    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:25.354372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.354347    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e3e93d0-3a9c-4823-b445-f1967d0244a2-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:25.355642 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.355616    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk" pod "4e3e93d0-3a9c-4823-b445-f1967d0244a2" (UID: "4e3e93d0-3a9c-4823-b445-f1967d0244a2"). InnerVolumeSpecName "kube-api-access-qhzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:25.355941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.355913    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials" pod "4e3e93d0-3a9c-4823-b445-f1967d0244a2" (UID: "4e3e93d0-3a9c-4823-b445-f1967d0244a2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:25.455619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.455545    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzjk\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-kube-api-access-qhzjk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:25.455619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.455571    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e3e93d0-3a9c-4823-b445-f1967d0244a2-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:25.565629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.565597    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:25.565788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.565648    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:25.761250 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:25.761168    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" podUID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" containerName="payload-processing" containerID="cri-o://3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e" gracePeriod=25
Oct 30 12:26:26.114905 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.114639    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" podUID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" containerName="payload-processing" containerID="cri-o://454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae" gracePeriod=25
Oct 30 12:26:26.119746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.119722    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:26:26.169204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.168699    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\") pod \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") "
Oct 30 12:26:26.169204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.168978    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\") pod \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") "
Oct 30 12:26:26.169204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.169074    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\") pod \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") "
Oct 30 12:26:26.169204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.169166    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\") pod \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\" (UID: \"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d\") "
Oct 30 12:26:26.170937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.170792    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume" pod "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" (UID: "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:26.175526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.171062    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume" pod "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" (UID: "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:26.175526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.174400    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials" pod "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" (UID: "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:26.175870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.175832    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v" pod "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" (UID: "bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d"). InnerVolumeSpecName "kube-api-access-bcn8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:26.181421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.181390    4453 generic.go:356] "Generic (PLEG): container finished" podID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" containerID="3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e" exitCode=0
Oct 30 12:26:26.181684 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.181671    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"
Oct 30 12:26:26.182623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.182240    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" event={"ID":"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d","Type":"ContainerDied","Data":"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"}
Oct 30 12:26:26.182623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.182425    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk" event={"ID":"bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d","Type":"ContainerDied","Data":"f5bf84891e79ddd1ce7e301897a34dde8cae8e63456b68f0d23999fb2c06b548"}
Oct 30 12:26:26.182623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.182461    4453 scope.go:122] "RemoveContainer" containerID="3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"
Oct 30 12:26:26.184216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.184197    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k" event={"ID":"4e3e93d0-3a9c-4823-b445-f1967d0244a2","Type":"ContainerDied","Data":"f35b18c61ecb57ff336aec47ff8a9b2134f3baaa0c72884b89a15d2f8a78f55b"}
Oct 30 12:26:26.184309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.184297    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"
Oct 30 12:26:26.240179 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.240150    4453 scope.go:122] "RemoveContainer" containerID="3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"
Oct 30 12:26:26.240525 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:26.240492    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e\": container with ID starting with 3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e not found: ID does not exist" containerID="3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"
Oct 30 12:26:26.240635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.240527    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e"} err="failed to get container status \"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e\": rpc error: code = NotFound desc = could not find container \"3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e\": container with ID starting with 3c72bbdfe062ea870d49c6957336b5984914200c31d30bfc10333f05ac4cf07e not found: ID does not exist"
Oct 30 12:26:26.240635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.240551    4453 scope.go:122] "RemoveContainer" containerID="65222af90981c3c5293e4450c01d100c98e9de66383829a55e853c57e632e6af"
Oct 30 12:26:26.269991 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.269964    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:26:26.272696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.272673    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.272696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.272695    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcn8v\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-kube-api-access-bcn8v\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.272910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.272706    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.272910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.272717    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.276372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.276341    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f8f132/payload-processing-external-model-e2e-extmod-a-f8f132-54cdrhczk"]
Oct 30 12:26:26.322650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.322538    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:26:26.328641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.328610    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-956cqk72k"]
Oct 30 12:26:26.507363 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.506225    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:26.566554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.566519    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:26.566751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.566575    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:26.577473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.577397    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\") pod \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") "
Oct 30 12:26:26.577627 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.577521    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\") pod \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") "
Oct 30 12:26:26.578524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.577832    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume" pod "0e887d3e-604b-4ed0-ad09-ad520c9f71c8" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:26.578524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.577943    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume" pod "0e887d3e-604b-4ed0-ad09-ad520c9f71c8" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:26:26.578524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.578172    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\") pod \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") "
Oct 30 12:26:26.578732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.578617    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\") pod \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\" (UID: \"0e887d3e-604b-4ed0-ad09-ad520c9f71c8\") "
Oct 30 12:26:26.579580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.579442    4453 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-routing-overlay-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.579580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.579469    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.580906 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.580879    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz" pod "0e887d3e-604b-4ed0-ad09-ad520c9f71c8" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8"). InnerVolumeSpecName "kube-api-access-74kbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:26.581309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.581146    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials" pod "0e887d3e-604b-4ed0-ad09-ad520c9f71c8" (UID: "0e887d3e-604b-4ed0-ad09-ad520c9f71c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:26:26.680469 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.680435    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74kbz\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-kube-api-access-74kbz\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.680469 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.680462    4453 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e887d3e-604b-4ed0-ad09-ad520c9f71c8-provider-credentials\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:26:26.952258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.952224    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" path="/var/lib/kubelet/pods/4e3e93d0-3a9c-4823-b445-f1967d0244a2/volumes"
Oct 30 12:26:26.953258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:26.953221    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" path="/var/lib/kubelet/pods/bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d/volumes"
Oct 30 12:26:27.218942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.218846    4453 generic.go:356] "Generic (PLEG): container finished" podID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" containerID="454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae" exitCode=0
Oct 30 12:26:27.218942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.218899    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"
Oct 30 12:26:27.218942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.218922    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" event={"ID":"0e887d3e-604b-4ed0-ad09-ad520c9f71c8","Type":"ContainerDied","Data":"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"}
Oct 30 12:26:27.219258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.218988    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c" event={"ID":"0e887d3e-604b-4ed0-ad09-ad520c9f71c8","Type":"ContainerDied","Data":"b1cf39509a3097d9397d01dd0111e428ba673d74a88e7db4bc457de62f158a52"}
Oct 30 12:26:27.219258 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.219020    4453 scope.go:122] "RemoveContainer" containerID="454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"
Oct 30 12:26:27.230350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.230310    4453 scope.go:122] "RemoveContainer" containerID="454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"
Oct 30 12:26:27.230606 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:27.230583    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae\": container with ID starting with 454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae not found: ID does not exist" containerID="454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"
Oct 30 12:26:27.230658 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.230616    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae"} err="failed to get container status \"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae\": rpc error: code = NotFound desc = could not find container \"454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae\": container with ID starting with 454429208c1587d54e631a91160332ff4940e8be02d5ac1a247b201b69ff8dae not found: ID does not exist"
Oct 30 12:26:27.266298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.266273    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:27.276011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.275989    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2bc4e2/payload-processing-external-model-e2e-extmod-b-2bc4e2-7d94vt45c"]
Oct 30 12:26:27.566322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.566241    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:27.566322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.566302    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:27.740189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.740157    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:26:27.741355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741305    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" containerName="payload-processing"
Oct 30 12:26:27.741355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741343    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741363    4453 cpu_manager.go:414] "containerMap: removing container" podUID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741369    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741376    4453 cpu_manager.go:414] "containerMap: removing container" podUID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741381    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741509    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741521    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3e93d0-3a9c-4823-b445-f1967d0244a2" containerName="payload-processing"
Oct 30 12:26:27.741552 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.741529    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3b9304-c193-4ea1-a5c6-e43e7a3efc4d" containerName="payload-processing"
Oct 30 12:26:27.747016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.746993    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.760771 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.760746    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-31a6e3-openshift-default-dockercfg-bdrpg\""
Oct 30 12:26:27.776464 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.776439    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:26:27.822827 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.822725    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82hs\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.822973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.822862    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.822973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.822936    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.822987    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.823024    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.823066    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823178 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.823161    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.823207    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.823316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.823301    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926188    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926261    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926300    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926357    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926446    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926482    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926564    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926665    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82hs\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.926985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.926780    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.976681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.928286    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.976681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.945085    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.976681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.961936    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.977256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.976847    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.977256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.977100    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.977961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.977935    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:27.996933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:27.985033    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:28.036355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.025957    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:28.036355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.034249    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82hs\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\") pod \"e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") " pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:28.069755 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.066651    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:28.517210 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:26:28.517176    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5809f1df_91c9_484a_b1b2_25c4434c5a82.slice/crio-20033cec34ee1f40205b25d65b4779485f42314e85bde481c5655f4104598b75 WatchSource:0}: Error finding container 20033cec34ee1f40205b25d65b4779485f42314e85bde481c5655f4104598b75: Status 404 returned error can't find the container with id 20033cec34ee1f40205b25d65b4779485f42314e85bde481c5655f4104598b75
Oct 30 12:26:28.519664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.519640    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:26:28.520465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.520432    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:28.520526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.520503    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:28.520568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.520536    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:26:28.568914 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.568835    4453 patch_prober.go:29] interesting pod/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 12:26:28.569240 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.569153    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 12:26:28.917241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:28.917195    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e887d3e-604b-4ed0-ad09-ad520c9f71c8" path="/var/lib/kubelet/pods/0e887d3e-604b-4ed0-ad09-ad520c9f71c8/volumes"
Oct 30 12:26:29.247461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.247368    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerStarted","Data":"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"}
Oct 30 12:26:29.247846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.247463    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerStarted","Data":"20033cec34ee1f40205b25d65b4779485f42314e85bde481c5655f4104598b75"}
Oct 30 12:26:29.300480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.300431    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podStartSLOduration=2.300417695 podStartE2EDuration="2.300417695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:29.295314348 +0000 UTC m=+3388.515599258" watchObservedRunningTime="2026-10-30 12:26:29.300417695 +0000 UTC m=+3388.520702606"
Oct 30 12:26:29.571590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.571500    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:26:29.571965 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.571946    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:26:29.572673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:29.572655    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:26:30.067149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:30.067118    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:30.073680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:30.073657    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:30.256938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:30.256909    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:30.257826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:30.257800    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:26:33.694572 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.694534    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:26:33.699506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.699484    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:33.719161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.719131    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:26:33.723647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.723623    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-31a6e3\""
Oct 30 12:26:33.723768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.723700    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-31a6e3-dockercfg-vdcs7\""
Oct 30 12:26:33.787084 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.787049    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:26:33.908953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.904274    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4cq\" (UniqueName: \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:33.908953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.905272    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:33.915379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.911580    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:26:33.915379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:33.912511    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.007233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.007126    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzzx\" (UniqueName: \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.007442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.007311    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:34.007525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.007503    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4cq\" (UniqueName: \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:34.007845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.007819    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.008231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.008212    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:34.028871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.028838    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4cq\" (UniqueName: \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\") pod \"payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:34.109561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.109511    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.109722 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.109619    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzzx\" (UniqueName: \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.110319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.110298    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.131238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.131200    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzzx\" (UniqueName: \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\") pod \"payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.244237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.244200    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:34.314832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.314804    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:34.554447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.554411    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:26:34.554846 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:26:34.554816    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872e9968_a3f9_4637_ac74_62eee04586d3.slice/crio-1e7b335cc92a0b3afe38cb1d8074c5021c9aae246a31953da858528714cc7564 WatchSource:0}: Error finding container 1e7b335cc92a0b3afe38cb1d8074c5021c9aae246a31953da858528714cc7564: Status 404 returned error can't find the container with id 1e7b335cc92a0b3afe38cb1d8074c5021c9aae246a31953da858528714cc7564
Oct 30 12:26:34.723845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:34.723816    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:26:34.725226 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:26:34.725192    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6a332d_96d7_4c32_9c7f_6212ff2a4c4b.slice/crio-ce886dfb1f54bfea4b602bb8c6d0614bd7fea6f26064c5941dacd1a8b9dfda7f WatchSource:0}: Error finding container ce886dfb1f54bfea4b602bb8c6d0614bd7fea6f26064c5941dacd1a8b9dfda7f: Status 404 returned error can't find the container with id ce886dfb1f54bfea4b602bb8c6d0614bd7fea6f26064c5941dacd1a8b9dfda7f
Oct 30 12:26:35.305230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.305195    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" event={"ID":"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b","Type":"ContainerStarted","Data":"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"}
Oct 30 12:26:35.305485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.305252    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" event={"ID":"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b","Type":"ContainerStarted","Data":"ce886dfb1f54bfea4b602bb8c6d0614bd7fea6f26064c5941dacd1a8b9dfda7f"}
Oct 30 12:26:35.305485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.305296    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:35.306968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.306945    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" event={"ID":"872e9968-a3f9-4637-ac74-62eee04586d3","Type":"ContainerStarted","Data":"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"}
Oct 30 12:26:35.307104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.306981    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" event={"ID":"872e9968-a3f9-4637-ac74-62eee04586d3","Type":"ContainerStarted","Data":"1e7b335cc92a0b3afe38cb1d8074c5021c9aae246a31953da858528714cc7564"}
Oct 30 12:26:35.307104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.307033    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:26:35.343980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.343928    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" podStartSLOduration=2.343915834 podStartE2EDuration="2.343915834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:35.340523615 +0000 UTC m=+3394.560808525" watchObservedRunningTime="2026-10-30 12:26:35.343915834 +0000 UTC m=+3394.564200744"
Oct 30 12:26:35.380803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:35.380757    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" podStartSLOduration=2.380743622 podStartE2EDuration="2.380743622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:26:35.376382167 +0000 UTC m=+3394.596667074" watchObservedRunningTime="2026-10-30 12:26:35.380743622 +0000 UTC m=+3394.601028531"
Oct 30 12:26:37.691353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:26:37.687277    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:26:46.317909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:46.317871    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:26:46.318366 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:26:46.317954    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:27:07.572654 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:27:07.572607    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:27:28.688761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.688724    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:27:28.693894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.693872    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.711048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.711027    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-31a6e3-serving-cert\""
Oct 30 12:27:28.725045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.725009    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:27:28.765142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.765087    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk746\" (UniqueName: \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.765370 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.765189    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.765370 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.765258    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.765711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.765690    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.867594 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.867559    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.867801 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.867632    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.867903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.867871    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.868376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.868353    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk746\" (UniqueName: \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.868531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.868510    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.870241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.870216    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.870362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.870303    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:28.889133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:28.889109    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk746\" (UniqueName: \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\") pod \"maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:29.012126 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:29.012030    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:29.328819 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:29.328786    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:27:29.330792 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:27:29.330760    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4362bb18_c55b_4326_8208_90b80ab4571f.slice/crio-8b1f69ba262faa703f14daf3eb1966aaa8b2c1675bdc0761cad350badb71f0f8 WatchSource:0}: Error finding container 8b1f69ba262faa703f14daf3eb1966aaa8b2c1675bdc0761cad350badb71f0f8: Status 404 returned error can't find the container with id 8b1f69ba262faa703f14daf3eb1966aaa8b2c1675bdc0761cad350badb71f0f8
Oct 30 12:27:30.032015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:30.031921    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" event={"ID":"4362bb18-c55b-4326-8208-90b80ab4571f","Type":"ContainerStarted","Data":"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"}
Oct 30 12:27:30.032015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:30.031984    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" event={"ID":"4362bb18-c55b-4326-8208-90b80ab4571f","Type":"ContainerStarted","Data":"8b1f69ba262faa703f14daf3eb1966aaa8b2c1675bdc0761cad350badb71f0f8"}
Oct 30 12:27:30.032015 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:30.032006    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:30.068486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:30.068428    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" podStartSLOduration=1.735363468 podStartE2EDuration="2.068415334s" totalImagesPullingTime="333.051866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:30.067993217 +0000 UTC m=+3449.288278128" watchObservedRunningTime="2026-10-30 12:27:30.068415334 +0000 UTC m=+3449.288700255"
Oct 30 12:27:35.593810 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.593768    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:27:35.598724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.598701    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.610997 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.610975    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a1f322-openshift-default-dockercfg-hxsqp\""
Oct 30 12:27:35.628635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.628604    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:27:35.691865 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.691826    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692038 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.691894    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj6h\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692045    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692093    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692208    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692394    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692509 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692428    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692565    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.692656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.692640    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.793947 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.793914    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794018    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794035    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794088    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794143    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794174    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794201    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj6h\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794289    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794316    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794421    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794658    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794710    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794892 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794869    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.794932 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.794875    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.796490 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.796458    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.796786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.796769    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.815209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.815187    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.815428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.815406    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj6h\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\") pod \"e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") " pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:35.914789 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:35.914762    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:36.045823 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:36.045787    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:27:36.238268 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:36.238239    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:27:36.239221 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:27:36.239197    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffea7c47_cd3e_4738_928c_3a680d656825.slice/crio-7c613e36d1c5cb3c55f023801d3b2e720e0ee9cc1922059ec86b8035b087eb30 WatchSource:0}: Error finding container 7c613e36d1c5cb3c55f023801d3b2e720e0ee9cc1922059ec86b8035b087eb30: Status 404 returned error can't find the container with id 7c613e36d1c5cb3c55f023801d3b2e720e0ee9cc1922059ec86b8035b087eb30
Oct 30 12:27:36.242489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:36.242458    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:36.242582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:36.242521    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:36.242582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:36.242551    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:37.149501 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.149463    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" event={"ID":"ffea7c47-cd3e-4738-928c-3a680d656825","Type":"ContainerStarted","Data":"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"}
Oct 30 12:27:37.149903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.149522    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" event={"ID":"ffea7c47-cd3e-4738-928c-3a680d656825","Type":"ContainerStarted","Data":"7c613e36d1c5cb3c55f023801d3b2e720e0ee9cc1922059ec86b8035b087eb30"}
Oct 30 12:27:37.201676 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.201617    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podStartSLOduration=2.201601769 podStartE2EDuration="2.201601769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:37.198591062 +0000 UTC m=+3456.418875983" watchObservedRunningTime="2026-10-30 12:27:37.201601769 +0000 UTC m=+3456.421886678"
Oct 30 12:27:37.573474 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:27:37.573396    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:27:37.916385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.915936    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:37.918314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.918287    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:37.918435 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:37.918372    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:38.915268 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:38.915229    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:38.915789 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:38.915292    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:39.858729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.858687    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:27:39.866841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.866809    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.879477 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.879452    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ae3538-openshift-default-dockercfg-kzncb\""
Oct 30 12:27:39.922567 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.922530    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:39.923028 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.922599    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:39.924032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.924006    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:27:39.980041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980007    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980093    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980306    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980438    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980523 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980458    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htr9b\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980584    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980810 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980791    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980897 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980832    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:39.980951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:39.980918    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.082587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082548    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.082795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082596    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.082795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082642    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.082795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082718    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.082958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082855    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.082978    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083025    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083038    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083053    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083064    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htr9b\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083126    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083142    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083547    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.083683 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.083657    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.085814 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.085792    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.086041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.086022    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.118398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.115065    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.118398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.116102    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htr9b\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\") pod \"e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") " pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.183419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.183385    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:40.584274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.584242    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:27:40.585894 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:27:40.585860    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2931ba8_16df_4903_bfa9_8cf801e80efe.slice/crio-3db3d70c4c4f52f4e4ee3f505d7465d58330ad8b97d754390442131f4691e4d5 WatchSource:0}: Error finding container 3db3d70c4c4f52f4e4ee3f505d7465d58330ad8b97d754390442131f4691e4d5: Status 404 returned error can't find the container with id 3db3d70c4c4f52f4e4ee3f505d7465d58330ad8b97d754390442131f4691e4d5
Oct 30 12:27:40.592362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.589769    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:40.592362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.589866    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:40.592362 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.589919    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:27:40.922909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.922873    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:40.923441 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:40.922935    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:41.184132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:41.184040    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" event={"ID":"f2931ba8-16df-4903-bfa9-8cf801e80efe","Type":"ContainerStarted","Data":"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"}
Oct 30 12:27:41.184132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:41.184100    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" event={"ID":"f2931ba8-16df-4903-bfa9-8cf801e80efe","Type":"ContainerStarted","Data":"3db3d70c4c4f52f4e4ee3f505d7465d58330ad8b97d754390442131f4691e4d5"}
Oct 30 12:27:41.224924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:41.224867    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podStartSLOduration=2.224852831 podStartE2EDuration="2.224852831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:27:41.224363344 +0000 UTC m=+3460.444648255" watchObservedRunningTime="2026-10-30 12:27:41.224852831 +0000 UTC m=+3460.445137740"
Oct 30 12:27:41.938358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:41.930447    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:41.938358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:41.930508    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:42.183541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:42.183508    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:42.185071 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:42.185032    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:42.185186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:42.185095    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:42.915467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:42.915424    4453 patch_prober.go:29] interesting pod/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 12:27:42.915685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:42.915484    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 12:27:43.202525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:43.200503    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:43.202525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:43.200574    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:43.921369 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:43.921284    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:43.921584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:43.921564    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:43.922313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:43.922296    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:27:44.184428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:44.184305    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:44.184428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:44.184394    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:45.184228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:45.184192    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:45.184659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:45.184255    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:46.184260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:46.184224    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:46.184690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:46.184276    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:47.184449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:47.184413    4453 patch_prober.go:29] interesting pod/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 12:27:47.184851 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:47.184469    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 12:27:48.189880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:48.189851    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:48.190322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:48.190268    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:27:48.190808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:27:48.190791    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:28:07.827357 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:28:07.823555    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:28:14.016918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:14.016873    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:28:14.018480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:14.017142    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy" containerID="cri-o://7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d" gracePeriod=30
Oct 30 12:28:19.324498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.324466    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:28:19.379085 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379049    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379197    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379248    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379288    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379554    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379584    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:19.379721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379609    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379640    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.379721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379679    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.380042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379736    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\") pod \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\" (UID: \"b2c9c68b-2298-4fc4-9d7f-55b32f86274e\") "
Oct 30 12:28:19.380042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379939    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:28:19.380042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.379978    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:19.380211 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380186    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:19.380650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380440    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:19.380775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380678    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.380775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380696    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.380775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380709    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.380775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380720    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.380775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.380732    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.383453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.383428    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:19.393374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.384021    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "kube-api-access-2mfqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:28:19.393374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.384233    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:28:19.393374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.388860    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo" pod "b2c9c68b-2298-4fc4-9d7f-55b32f86274e" (UID: "b2c9c68b-2298-4fc4-9d7f-55b32f86274e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:28:19.482452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.482421    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.482452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.482448    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.482452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.482458    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfqw\" (UniqueName: \"kubernetes.io/projected/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-kube-api-access-2mfqw\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.482680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.482467    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2c9c68b-2298-4fc4-9d7f-55b32f86274e-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:28:19.930722 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.930683    4453 generic.go:356] "Generic (PLEG): container finished" podID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerID="7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d" exitCode=0
Oct 30 12:28:19.931672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.930742    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"
Oct 30 12:28:19.931672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.930784    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" event={"ID":"b2c9c68b-2298-4fc4-9d7f-55b32f86274e","Type":"ContainerDied","Data":"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"}
Oct 30 12:28:19.931672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.930856    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx" event={"ID":"b2c9c68b-2298-4fc4-9d7f-55b32f86274e","Type":"ContainerDied","Data":"cf1d897efb0dfa0adfd071b9bb059576eeeaceb254c15bf3517f79e2a37f65d1"}
Oct 30 12:28:19.931672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.930889    4453 scope.go:122] "RemoveContainer" containerID="7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"
Oct 30 12:28:19.943766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.943748    4453 scope.go:122] "RemoveContainer" containerID="7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"
Oct 30 12:28:19.944044 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:28:19.944025    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d\": container with ID starting with 7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d not found: ID does not exist" containerID="7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"
Oct 30 12:28:19.944112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.944051    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d"} err="failed to get container status \"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d\": rpc error: code = NotFound desc = could not find container \"7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d\": container with ID starting with 7022fafd8cf20792ef2a0b1423f8651784b5e2bd1c76aa23a5983324111cc16d not found: ID does not exist"
Oct 30 12:28:19.989968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.989937    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:28:19.996297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:19.996274    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-737f64-openshift-default-6ddddc9965-4d5lx"]
Oct 30 12:28:20.914985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.914953    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" path="/var/lib/kubelet/pods/b2c9c68b-2298-4fc4-9d7f-55b32f86274e/volumes"
Oct 30 12:28:20.940811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.940781    4453 generic.go:356] "Generic (PLEG): container finished" podID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerID="b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca" exitCode=0
Oct 30 12:28:20.940980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.940852    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerDied","Data":"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"}
Oct 30 12:28:20.941382 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.941357    4453 scope.go:122] "RemoveContainer" containerID="b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"
Oct 30 12:28:20.943496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.943464    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:28:20.943569 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.943559    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:28:20.943611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:20.943595    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:28:21.955352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:21.952600    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerStarted","Data":"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"}
Oct 30 12:28:22.067781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:22.067747    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:28:22.067973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:22.067935    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:22.068021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:22.067987    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:23.067630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:23.067597    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:23.068038 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:23.067648    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:24.068144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:24.068103    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:24.068675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:24.068165    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:25.067939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:25.067898    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:25.068131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:25.067953    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:26.067285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:26.067245    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:26.067710 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:26.067307    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:27.067640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:27.067605    4453 patch_prober.go:29] interesting pod/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 12:28:27.068040 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:27.067666    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 12:28:28.073234 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:28.073193    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:28:28.073792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:28.073761    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:28:28.074522 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:28.074504    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:28:37.090826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.090793    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:28:37.091699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.091681    4453 cpu_manager.go:414] "containerMap: removing container" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy"
Oct 30 12:28:37.091772 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.091701    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy"
Oct 30 12:28:37.091826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.091817    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c9c68b-2298-4fc4-9d7f-55b32f86274e" containerName="istio-proxy"
Oct 30 12:28:37.095732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.095704    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.113347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.113313    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ffea51\""
Oct 30 12:28:37.127740 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.127715    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:28:37.155453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.155423    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:28:37.161412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.161390    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.178186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.178155    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ffea51-dockercfg-2lrmk\""
Oct 30 12:28:37.179102 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.179071    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.179242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.179144    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqp6\" (UniqueName: \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.184472 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.184442    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:28:37.383793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.383754    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.383995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.383844    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqp6\" (UniqueName: \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.389204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.389176    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.444721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.444686    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqp6\" (UniqueName: \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\") pod \"payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.485197 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.485158    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86577\" (UniqueName: \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.485383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.485358    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.574066 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:28:37.574026    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:28:37.587108 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.587075    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.587384 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.587318    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86577\" (UniqueName: \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.587937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.587916    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.607845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.607821    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86577\" (UniqueName: \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\") pod \"payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:37.717057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.716981    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:37.779236 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:37.779208    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:38.021750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.021722    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:28:38.022304 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:28:38.022281    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758b1d1b_3cda_4f5a_af48_453121857572.slice/crio-f5367bc33e30aa32b63999eca82908c83c7f1f2800aa995fd64f4e474fa9fe64 WatchSource:0}: Error finding container f5367bc33e30aa32b63999eca82908c83c7f1f2800aa995fd64f4e474fa9fe64: Status 404 returned error can't find the container with id f5367bc33e30aa32b63999eca82908c83c7f1f2800aa995fd64f4e474fa9fe64
Oct 30 12:28:38.108676 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.108649    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:28:38.109087 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:28:38.108880    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fd811c_b82c_4af1_96a7_0eff3b3aac2d.slice/crio-47fe5e239a8d2e24d860c42aa7ca24ec9bcd2f1f5bcbcaab45916e2cf9765556 WatchSource:0}: Error finding container 47fe5e239a8d2e24d860c42aa7ca24ec9bcd2f1f5bcbcaab45916e2cf9765556: Status 404 returned error can't find the container with id 47fe5e239a8d2e24d860c42aa7ca24ec9bcd2f1f5bcbcaab45916e2cf9765556
Oct 30 12:28:38.118266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.118232    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" event={"ID":"758b1d1b-3cda-4f5a-af48-453121857572","Type":"ContainerStarted","Data":"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"}
Oct 30 12:28:38.118405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.118294    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" event={"ID":"758b1d1b-3cda-4f5a-af48-453121857572","Type":"ContainerStarted","Data":"f5367bc33e30aa32b63999eca82908c83c7f1f2800aa995fd64f4e474fa9fe64"}
Oct 30 12:28:38.119280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.118699    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:38.162768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:38.162723    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" podStartSLOduration=1.162708689 podStartE2EDuration="1.162708689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:38.154112557 +0000 UTC m=+3517.374397489" watchObservedRunningTime="2026-10-30 12:28:38.162708689 +0000 UTC m=+3517.382993599"
Oct 30 12:28:39.136877 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:39.136842    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" event={"ID":"73fd811c-b82c-4af1-96a7-0eff3b3aac2d","Type":"ContainerStarted","Data":"ca3b89908dab064384660469e16e5cbf817bf26909e62fb19bb3e57770cd9503"}
Oct 30 12:28:39.137283 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:39.136892    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" event={"ID":"73fd811c-b82c-4af1-96a7-0eff3b3aac2d","Type":"ContainerStarted","Data":"47fe5e239a8d2e24d860c42aa7ca24ec9bcd2f1f5bcbcaab45916e2cf9765556"}
Oct 30 12:28:39.137283 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:39.136925    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:28:39.182738 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:39.182685    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" podStartSLOduration=2.182669983 podStartE2EDuration="2.182669983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:28:39.176034902 +0000 UTC m=+3518.396319816" watchObservedRunningTime="2026-10-30 12:28:39.182669983 +0000 UTC m=+3518.402954893"
Oct 30 12:28:49.139602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:49.139567    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:28:51.158671 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:28:51.158644    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:29:03.083660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:03.083625    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:29:03.084188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:03.083816    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy" containerID="cri-o://69861f7662cd4c21b00bf285272225d5a6f4f9edc083d8f8aeac879a8ac5712a" gracePeriod=30
Oct 30 12:29:07.572420 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:29:07.572381    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:29:08.392411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.392372    4453 generic.go:356] "Generic (PLEG): container finished" podID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerID="69861f7662cd4c21b00bf285272225d5a6f4f9edc083d8f8aeac879a8ac5712a" exitCode=0
Oct 30 12:29:08.392575 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.392436    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" event={"ID":"768b7c4b-6c3f-46bc-bab4-350c558127c2","Type":"ContainerDied","Data":"69861f7662cd4c21b00bf285272225d5a6f4f9edc083d8f8aeac879a8ac5712a"}
Oct 30 12:29:08.439106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.439080    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:29:08.570223 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570190    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570275    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570377    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570548 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570533    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570621 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570604    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570679    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570753 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570726    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570755    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570870 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570815    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\") pod \"768b7c4b-6c3f-46bc-bab4-350c558127c2\" (UID: \"768b7c4b-6c3f-46bc-bab4-350c558127c2\") "
Oct 30 12:29:08.570927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570892    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.571047 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.570931    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:29:08.571124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571089    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.571124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571113    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.571243 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571207    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.571706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571681    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768b7c4b-6c3f-46bc-bab4-350c558127c2-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.571799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571712    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.571799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571727    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.571799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571742    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.571799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.571757    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.572700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.572680    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:29:08.573162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.572900    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:08.573162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.573043    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:29:08.573162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.573116    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5" pod "768b7c4b-6c3f-46bc-bab4-350c558127c2" (UID: "768b7c4b-6c3f-46bc-bab4-350c558127c2"). InnerVolumeSpecName "kube-api-access-v9dz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:29:08.672867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.672784    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.672867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.672816    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.672867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.672828    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:08.672867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:08.672839    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dz5\" (UniqueName: \"kubernetes.io/projected/768b7c4b-6c3f-46bc-bab4-350c558127c2-kube-api-access-v9dz5\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:29:09.401407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:09.401380    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"
Oct 30 12:29:09.401592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:09.401375    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z" event={"ID":"768b7c4b-6c3f-46bc-bab4-350c558127c2","Type":"ContainerDied","Data":"f1f24125685a869a275e379fee9623ebd54a753be4740dea37b3b3891b124b03"}
Oct 30 12:29:09.401592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:09.401511    4453 scope.go:122] "RemoveContainer" containerID="69861f7662cd4c21b00bf285272225d5a6f4f9edc083d8f8aeac879a8ac5712a"
Oct 30 12:29:09.457375 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:09.457347    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:29:09.466659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:09.466634    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2924z"]
Oct 30 12:29:10.914564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:29:10.914529    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" path="/var/lib/kubelet/pods/768b7c4b-6c3f-46bc-bab4-350c558127c2/volumes"
Oct 30 12:29:37.573464 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:29:37.573418    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:30:00.225189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.225149    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:00.226591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.226565    4453 cpu_manager.go:414] "containerMap: removing container" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy"
Oct 30 12:30:00.226591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.226592    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy"
Oct 30 12:30:00.226830 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.226813    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="768b7c4b-6c3f-46bc-bab4-350c558127c2" containerName="istio-proxy"
Oct 30 12:30:00.234705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.234678    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:00.234871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.234850    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.241365 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.241302    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:00.241641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.241625    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.250773 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.250753    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.266702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.265418    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:00.272820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.272325    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:00.273562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.273262    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284350    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284525    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhb74\" (UniqueName: \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284668    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284694    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqh99\" (UniqueName: \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284728    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4m9\" (UniqueName: \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.285871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.284855    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.293136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.291637    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:00.293136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.291684    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:00.310629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.304421    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.312624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.311626    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:00.312624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.311673    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:00.312624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.311815    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.335379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.331502    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:00.335379 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.331555    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:00.337255 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.337230    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.343726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.341077    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:00.343726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.341120    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:00.344164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.344144    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.350384 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.348719    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:00.350384 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.348813    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.350384 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.349450    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:00.377007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.376974    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.387852    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.387966    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389044    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389186    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389236    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhrs\" (UniqueName: \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389387    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389476    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbdk\" (UniqueName: \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389522    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhb74\" (UniqueName: \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389799    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389845    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.389872    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.390628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.390222    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.390797    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.390607    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqh99\" (UniqueName: \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.390958    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skc64\" (UniqueName: \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.391000    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4m9\" (UniqueName: \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.391044    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.392295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.391211    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6f4\" (UniqueName: \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.392678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.392316    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.392818 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.392797    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.404405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.404309    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:00.409937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.409712    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:00.410884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.410857    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhb74\" (UniqueName: \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\") pod \"maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.411639 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.411615    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqh99\" (UniqueName: \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\") pod \"maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.412630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.412023    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4m9\" (UniqueName: \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\") pod \"maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.492644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492569    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.492644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492622    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhrs\" (UniqueName: \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.492850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492651    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.492850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492700    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.492850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492786    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbdk\" (UniqueName: \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.493053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492898    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.493053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.492926    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.493053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493021    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skc64\" (UniqueName: \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.493223 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493070    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.493223 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493114    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6f4\" (UniqueName: \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.493223 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493172    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.493437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493300    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpck4\" (UniqueName: \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.493437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493303    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.493642 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493617    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.493712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493637    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.493786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493766    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.493927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.493898    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.513807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.513779    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skc64\" (UniqueName: \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\") pod \"maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.513948 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.513878    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhrs\" (UniqueName: \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\") pod \"maas-api-key-cleanup-29889390-gl87v\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.514025 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.513960    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\") pod \"maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.514161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.514140    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbdk\" (UniqueName: \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.514221 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.514166    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6f4\" (UniqueName: \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\") pod \"maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.571443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.571411    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:00.579208 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.579185    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:00.595141 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.595109    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.595459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.595434    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpck4\" (UniqueName: \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.595875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.595853    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.596034 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.596018    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:00.618002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.617973    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpck4\" (UniqueName: \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\") pod \"maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.641242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.640969    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:00.678032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.677539    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:00.701564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.701362    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:00.713802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.712119    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:00.751729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.751586    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:00.757031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.756705    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:00.907749 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:00.903368    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999feecb_154b_4d75_85fd_08ea8fc965e8.slice/crio-d3124f45f216469648c299be74b319d3cc7789ceb1ca8eb8575b7a267b8d8fd8 WatchSource:0}: Error finding container d3124f45f216469648c299be74b319d3cc7789ceb1ca8eb8575b7a267b8d8fd8: Status 404 returned error can't find the container with id d3124f45f216469648c299be74b319d3cc7789ceb1ca8eb8575b7a267b8d8fd8
Oct 30 12:30:00.912541 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:00.912517    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5dad6e_ec23_4f28_a550_cd49b2f7bc05.slice/crio-60c9816fa06a3e77e19d415d696dc59c8b88d4b3b60ad00a6eff0952d4c44e03 WatchSource:0}: Error finding container 60c9816fa06a3e77e19d415d696dc59c8b88d4b3b60ad00a6eff0952d4c44e03: Status 404 returned error can't find the container with id 60c9816fa06a3e77e19d415d696dc59c8b88d4b3b60ad00a6eff0952d4c44e03
Oct 30 12:30:00.927030 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.926985    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:00.927242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:00.927217    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:01.139921 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.139891    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:01.191302 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.191275    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:01.202051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.202027    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:01.212908 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.212883    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:01.252914 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.252889    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:01.291525 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:01.291476    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cfa747_2078_4af2_97c2_0d3e02a8e8ed.slice/crio-fd8d53c8e60c8b3644364fe5e7f68d96cea4f90490502482cb20d1540c73b487 WatchSource:0}: Error finding container fd8d53c8e60c8b3644364fe5e7f68d96cea4f90490502482cb20d1540c73b487: Status 404 returned error can't find the container with id fd8d53c8e60c8b3644364fe5e7f68d96cea4f90490502482cb20d1540c73b487
Oct 30 12:30:01.294302 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:01.292726    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec590f6_bd3d_4f76_8ff1_b7cac7cc1b0a.slice/crio-f1b90d692fd761b1e45cbd2f6941030bf95ee4d705b94b84c0722365efb8514a WatchSource:0}: Error finding container f1b90d692fd761b1e45cbd2f6941030bf95ee4d705b94b84c0722365efb8514a: Status 404 returned error can't find the container with id f1b90d692fd761b1e45cbd2f6941030bf95ee4d705b94b84c0722365efb8514a
Oct 30 12:30:01.295964 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:01.295922    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f06ab_42af_4d39_9776_f4f7d3009c2c.slice/crio-366ccdc737d26429acd4d6bbcedeb154648345c0758131226c85964ba518fd8f WatchSource:0}: Error finding container 366ccdc737d26429acd4d6bbcedeb154648345c0758131226c85964ba518fd8f: Status 404 returned error can't find the container with id 366ccdc737d26429acd4d6bbcedeb154648345c0758131226c85964ba518fd8f
Oct 30 12:30:01.338660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.338603    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:01.342608 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:01.342576    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b3d23f_5af0_418b_9542_6d555273f0a5.slice/crio-286bf3080d13881a3e820a76bf3fea9da3364a6aa422098e5c09b0fc7e0e96b8 WatchSource:0}: Error finding container 286bf3080d13881a3e820a76bf3fea9da3364a6aa422098e5c09b0fc7e0e96b8: Status 404 returned error can't find the container with id 286bf3080d13881a3e820a76bf3fea9da3364a6aa422098e5c09b0fc7e0e96b8
Oct 30 12:30:01.346266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.346237    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:01.353775 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:30:01.353753    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051a1c38_ecee_484c_bf0a_52dc4783379a.slice/crio-537d86dc8db898b8a83f5e2dbbcdc01e04e351f0ff36c48a89aa1365a3d4ac98 WatchSource:0}: Error finding container 537d86dc8db898b8a83f5e2dbbcdc01e04e351f0ff36c48a89aa1365a3d4ac98: Status 404 returned error can't find the container with id 537d86dc8db898b8a83f5e2dbbcdc01e04e351f0ff36c48a89aa1365a3d4ac98
Oct 30 12:30:01.937267 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.937222    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerStarted","Data":"4b44e093635ab05d972db04878dda85c3684028242fa1d84e51a3f629d97bbbc"}
Oct 30 12:30:01.937732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.937311    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerStarted","Data":"60c9816fa06a3e77e19d415d696dc59c8b88d4b3b60ad00a6eff0952d4c44e03"}
Oct 30 12:30:01.939405 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.939375    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerStarted","Data":"117595a1f38d7c03cedb80c3cbb595c12b950430a1dbd91221cf33037f7ab6fd"}
Oct 30 12:30:01.939520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.939425    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerStarted","Data":"d3124f45f216469648c299be74b319d3cc7789ceb1ca8eb8575b7a267b8d8fd8"}
Oct 30 12:30:01.941084 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.941058    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerStarted","Data":"366ccdc737d26429acd4d6bbcedeb154648345c0758131226c85964ba518fd8f"}
Oct 30 12:30:01.943937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.943910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" event={"ID":"3aed1b94-6670-47f0-92eb-5d340291567f","Type":"ContainerStarted","Data":"7157106cf22ebe74cf91b673b87b72f3799383fc0134e6a5c54e87548ac9595e"}
Oct 30 12:30:01.945613 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.945587    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" event={"ID":"051a1c38-ecee-484c-bf0a-52dc4783379a","Type":"ContainerStarted","Data":"537d86dc8db898b8a83f5e2dbbcdc01e04e351f0ff36c48a89aa1365a3d4ac98"}
Oct 30 12:30:01.947099 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.947068    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerStarted","Data":"286bf3080d13881a3e820a76bf3fea9da3364a6aa422098e5c09b0fc7e0e96b8"}
Oct 30 12:30:01.948743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.948722    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerStarted","Data":"1c63c792e0fdd4bfde7226cfcabb6bb4f1b696bcd6971eb29d30f1bfa14df6aa"}
Oct 30 12:30:01.950606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.950586    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerStarted","Data":"cce3a2f9b363e4b3643c3aab8616ee3670d46922182438d0394ec1770fe5772b"}
Oct 30 12:30:01.950688 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.950629    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerStarted","Data":"f1b90d692fd761b1e45cbd2f6941030bf95ee4d705b94b84c0722365efb8514a"}
Oct 30 12:30:01.951878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.951855    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerStarted","Data":"fd8d53c8e60c8b3644364fe5e7f68d96cea4f90490502482cb20d1540c73b487"}
Oct 30 12:30:01.974959 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:01.974897    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" podStartSLOduration=1.974880115 podStartE2EDuration="1.974880115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:01.97237 +0000 UTC m=+3601.192654904" watchObservedRunningTime="2026-10-30 12:30:01.974880115 +0000 UTC m=+3601.195165025"
Oct 30 12:30:02.013452 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.013397    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" podStartSLOduration=2.013380705 podStartE2EDuration="2.013380705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.008152152 +0000 UTC m=+3601.228437063" watchObservedRunningTime="2026-10-30 12:30:02.013380705 +0000 UTC m=+3601.233665616"
Oct 30 12:30:02.045144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.045094    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" podStartSLOduration=2.045081335 podStartE2EDuration="2.045081335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:02.042209777 +0000 UTC m=+3601.262494687" watchObservedRunningTime="2026-10-30 12:30:02.045081335 +0000 UTC m=+3601.265366244"
Oct 30 12:30:02.961159 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.961071    4453 generic.go:356] "Generic (PLEG): container finished" podID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerID="8028ff86995e6f95208ee1136519b7b1e44dc0c09ab7eeb4c0739f41ba6f3ef7" exitCode=60
Oct 30 12:30:02.961636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.961158    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" event={"ID":"051a1c38-ecee-484c-bf0a-52dc4783379a","Type":"ContainerDied","Data":"8028ff86995e6f95208ee1136519b7b1e44dc0c09ab7eeb4c0739f41ba6f3ef7"}
Oct 30 12:30:02.961636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.961540    4453 scope.go:122] "RemoveContainer" containerID="8028ff86995e6f95208ee1136519b7b1e44dc0c09ab7eeb4c0739f41ba6f3ef7"
Oct 30 12:30:02.963114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.963088    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerID="4d896c85b1965ebafda2009b4aad0461c89d23261228b682be6e66b2867de8c9" exitCode=60
Oct 30 12:30:02.963231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.963136    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerDied","Data":"4d896c85b1965ebafda2009b4aad0461c89d23261228b682be6e66b2867de8c9"}
Oct 30 12:30:02.963637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.963518    4453 scope.go:122] "RemoveContainer" containerID="4d896c85b1965ebafda2009b4aad0461c89d23261228b682be6e66b2867de8c9"
Oct 30 12:30:02.967645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.967622    4453 generic.go:356] "Generic (PLEG): container finished" podID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerID="6c7da1a61e084e41c1b4d206cef032802529299884ef0927d38f278c61e187be" exitCode=60
Oct 30 12:30:02.967735 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.967697    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerDied","Data":"6c7da1a61e084e41c1b4d206cef032802529299884ef0927d38f278c61e187be"}
Oct 30 12:30:02.967980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.967962    4453 scope.go:122] "RemoveContainer" containerID="6c7da1a61e084e41c1b4d206cef032802529299884ef0927d38f278c61e187be"
Oct 30 12:30:02.969719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.969700    4453 generic.go:356] "Generic (PLEG): container finished" podID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerID="cce3a2f9b363e4b3643c3aab8616ee3670d46922182438d0394ec1770fe5772b" exitCode=60
Oct 30 12:30:02.969812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.969784    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerDied","Data":"cce3a2f9b363e4b3643c3aab8616ee3670d46922182438d0394ec1770fe5772b"}
Oct 30 12:30:02.970090 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.970073    4453 scope.go:122] "RemoveContainer" containerID="cce3a2f9b363e4b3643c3aab8616ee3670d46922182438d0394ec1770fe5772b"
Oct 30 12:30:02.976060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.972055    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerID="0bb4f8bec8c65ee8152018032917b500be890cf4ffef40c4528ecf700fec1086" exitCode=60
Oct 30 12:30:02.976060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.972183    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerDied","Data":"0bb4f8bec8c65ee8152018032917b500be890cf4ffef40c4528ecf700fec1086"}
Oct 30 12:30:02.976060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.972523    4453 scope.go:122] "RemoveContainer" containerID="0bb4f8bec8c65ee8152018032917b500be890cf4ffef40c4528ecf700fec1086"
Oct 30 12:30:02.978088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.978064    4453 generic.go:356] "Generic (PLEG): container finished" podID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerID="4b44e093635ab05d972db04878dda85c3684028242fa1d84e51a3f629d97bbbc" exitCode=60
Oct 30 12:30:02.978195 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.978129    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerDied","Data":"4b44e093635ab05d972db04878dda85c3684028242fa1d84e51a3f629d97bbbc"}
Oct 30 12:30:02.978435 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.978421    4453 scope.go:122] "RemoveContainer" containerID="4b44e093635ab05d972db04878dda85c3684028242fa1d84e51a3f629d97bbbc"
Oct 30 12:30:02.980694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.980645    4453 generic.go:356] "Generic (PLEG): container finished" podID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerID="117595a1f38d7c03cedb80c3cbb595c12b950430a1dbd91221cf33037f7ab6fd" exitCode=60
Oct 30 12:30:02.980763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.980718    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerDied","Data":"117595a1f38d7c03cedb80c3cbb595c12b950430a1dbd91221cf33037f7ab6fd"}
Oct 30 12:30:02.980961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.980939    4453 scope.go:122] "RemoveContainer" containerID="117595a1f38d7c03cedb80c3cbb595c12b950430a1dbd91221cf33037f7ab6fd"
Oct 30 12:30:02.982424 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.982403    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerID="b9a8dbdda8a2992c759e65731d9f62dcf204a90c26ddf405aedb48398bb0c5af" exitCode=60
Oct 30 12:30:02.982521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.982431    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerDied","Data":"b9a8dbdda8a2992c759e65731d9f62dcf204a90c26ddf405aedb48398bb0c5af"}
Oct 30 12:30:02.982706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.982690    4453 scope.go:122] "RemoveContainer" containerID="b9a8dbdda8a2992c759e65731d9f62dcf204a90c26ddf405aedb48398bb0c5af"
Oct 30 12:30:02.984224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.984201    4453 generic.go:356] "Generic (PLEG): container finished" podID="3aed1b94-6670-47f0-92eb-5d340291567f" containerID="325fdff33ad36a146868d29e746951f169200fc54f48672f885f4ddbc2bc8f5a" exitCode=60
Oct 30 12:30:02.984316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.984280    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" event={"ID":"3aed1b94-6670-47f0-92eb-5d340291567f","Type":"ContainerDied","Data":"325fdff33ad36a146868d29e746951f169200fc54f48672f885f4ddbc2bc8f5a"}
Oct 30 12:30:02.984483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:02.984470    4453 scope.go:122] "RemoveContainer" containerID="325fdff33ad36a146868d29e746951f169200fc54f48672f885f4ddbc2bc8f5a"
Oct 30 12:30:03.999655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:03.999619    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerStarted","Data":"0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"}
Oct 30 12:30:04.002122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:04.002089    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerStarted","Data":"c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"}
Oct 30 12:30:04.007191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:04.006240    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerStarted","Data":"22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"}
Oct 30 12:30:04.037684 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:04.037638    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" podStartSLOduration=4.037626072 podStartE2EDuration="4.037626072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.034299421 +0000 UTC m=+3603.254584331" watchObservedRunningTime="2026-10-30 12:30:04.037626072 +0000 UTC m=+3603.257910981"
Oct 30 12:30:04.068165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:04.068115    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" podStartSLOduration=4.068100749 podStartE2EDuration="4.068100749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:04.067839521 +0000 UTC m=+3603.288124429" watchObservedRunningTime="2026-10-30 12:30:04.068100749 +0000 UTC m=+3603.288385671"
Oct 30 12:30:05.015680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.015653    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerStarted","Data":"f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"}
Oct 30 12:30:05.017506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.017479    4453 generic.go:356] "Generic (PLEG): container finished" podID="3aed1b94-6670-47f0-92eb-5d340291567f" containerID="c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6" exitCode=60
Oct 30 12:30:05.017593 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.017556    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" event={"ID":"3aed1b94-6670-47f0-92eb-5d340291567f","Type":"ContainerDied","Data":"c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6"}
Oct 30 12:30:05.017642 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.017614    4453 scope.go:122] "RemoveContainer" containerID="325fdff33ad36a146868d29e746951f169200fc54f48672f885f4ddbc2bc8f5a"
Oct 30 12:30:05.017829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.017806    4453 scope.go:122] "RemoveContainer" containerID="c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6"
Oct 30 12:30:05.018168 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.018147    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244_odh-ai-gateway-infra(3aed1b94-6670-47f0-92eb-5d340291567f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" podUID="3aed1b94-6670-47f0-92eb-5d340291567f"
Oct 30 12:30:05.019826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.019806    4453 generic.go:356] "Generic (PLEG): container finished" podID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerID="85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05" exitCode=60
Oct 30 12:30:05.019914 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.019889    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" event={"ID":"051a1c38-ecee-484c-bf0a-52dc4783379a","Type":"ContainerDied","Data":"85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05"}
Oct 30 12:30:05.020134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.020116    4453 scope.go:122] "RemoveContainer" containerID="85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05"
Oct 30 12:30:05.020455 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.020434    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc_odh-ai-gateway-infra(051a1c38-ecee-484c-bf0a-52dc4783379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a"
Oct 30 12:30:05.021806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.021778    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerID="0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25" exitCode=60
Oct 30 12:30:05.021892 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.021860    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerDied","Data":"0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"}
Oct 30 12:30:05.022074 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.022053    4453 scope.go:122] "RemoveContainer" containerID="0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"
Oct 30 12:30:05.022370 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.022316    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9_odh-ai-gateway-infra(e0b3d23f-5af0-418b-9542-6d555273f0a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5"
Oct 30 12:30:05.023819 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.023798    4453 generic.go:356] "Generic (PLEG): container finished" podID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerID="7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42" exitCode=60
Oct 30 12:30:05.023934 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.023863    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerDied","Data":"7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42"}
Oct 30 12:30:05.024055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.024040    4453 scope.go:122] "RemoveContainer" containerID="7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42"
Oct 30 12:30:05.024800 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.024288    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279_odh-ai-gateway-infra(52ed9bc8-5a69-44d2-87e9-6f3b15be19d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9"
Oct 30 12:30:05.026296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.026273    4453 generic.go:356] "Generic (PLEG): container finished" podID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerID="c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942" exitCode=60
Oct 30 12:30:05.026421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.026390    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerDied","Data":"c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"}
Oct 30 12:30:05.026561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.026547    4453 scope.go:122] "RemoveContainer" containerID="c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"
Oct 30 12:30:05.026806 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.026779    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk_odh-ai-gateway-infra(1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a"
Oct 30 12:30:05.028396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.028373    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerID="22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001" exitCode=60
Oct 30 12:30:05.028473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.028452    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerDied","Data":"22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"}
Oct 30 12:30:05.028656 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.028628    4453 scope.go:122] "RemoveContainer" containerID="22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"
Oct 30 12:30:05.028905 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.028883    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g_odh-ai-gateway-infra(e0cfa747-2078-4af2-97c2-0d3e02a8e8ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed"
Oct 30 12:30:05.035210 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.035187    4453 generic.go:356] "Generic (PLEG): container finished" podID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerID="6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11" exitCode=60
Oct 30 12:30:05.035311 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.035273    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerDied","Data":"6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11"}
Oct 30 12:30:05.035496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.035479    4453 scope.go:122] "RemoveContainer" containerID="6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11"
Oct 30 12:30:05.035837 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:05.035722    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg_odh-ai-gateway-infra(0c5dad6e-ec23-4f28-a550-cd49b2f7bc05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05"
Oct 30 12:30:05.035893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.035866    4453 scope.go:122] "RemoveContainer" containerID="8028ff86995e6f95208ee1136519b7b1e44dc0c09ab7eeb4c0739f41ba6f3ef7"
Oct 30 12:30:05.037516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.037492    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerStarted","Data":"a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"}
Oct 30 12:30:05.050011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.049995    4453 scope.go:122] "RemoveContainer" containerID="4d896c85b1965ebafda2009b4aad0461c89d23261228b682be6e66b2867de8c9"
Oct 30 12:30:05.053728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.053689    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" podStartSLOduration=5.053678655 podStartE2EDuration="5.053678655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:05.049392263 +0000 UTC m=+3604.269677174" watchObservedRunningTime="2026-10-30 12:30:05.053678655 +0000 UTC m=+3604.273963564"
Oct 30 12:30:05.064531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.064509    4453 scope.go:122] "RemoveContainer" containerID="6c7da1a61e084e41c1b4d206cef032802529299884ef0927d38f278c61e187be"
Oct 30 12:30:05.078071 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.078051    4453 scope.go:122] "RemoveContainer" containerID="cce3a2f9b363e4b3643c3aab8616ee3670d46922182438d0394ec1770fe5772b"
Oct 30 12:30:05.264121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.264090    4453 scope.go:122] "RemoveContainer" containerID="0bb4f8bec8c65ee8152018032917b500be890cf4ffef40c4528ecf700fec1086"
Oct 30 12:30:05.316750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:05.316721    4453 scope.go:122] "RemoveContainer" containerID="4b44e093635ab05d972db04878dda85c3684028242fa1d84e51a3f629d97bbbc"
Oct 30 12:30:06.048706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.048630    4453 scope.go:122] "RemoveContainer" containerID="6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11"
Oct 30 12:30:06.049081 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.048993    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg_odh-ai-gateway-infra(0c5dad6e-ec23-4f28-a550-cd49b2f7bc05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05"
Oct 30 12:30:06.050302 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.050276    4453 generic.go:356] "Generic (PLEG): container finished" podID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerID="a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c" exitCode=60
Oct 30 12:30:06.050457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.050356    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerDied","Data":"a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"}
Oct 30 12:30:06.050457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.050407    4453 scope.go:122] "RemoveContainer" containerID="117595a1f38d7c03cedb80c3cbb595c12b950430a1dbd91221cf33037f7ab6fd"
Oct 30 12:30:06.050677 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.050644    4453 scope.go:122] "RemoveContainer" containerID="a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"
Oct 30 12:30:06.051030 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.051003    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl_odh-ai-gateway-infra(999feecb-154b-4d75-85fd-08ea8fc965e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8"
Oct 30 12:30:06.052524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.052502    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerID="f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066" exitCode=60
Oct 30 12:30:06.052618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.052585    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerDied","Data":"f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"}
Oct 30 12:30:06.052800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.052784    4453 scope.go:122] "RemoveContainer" containerID="f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"
Oct 30 12:30:06.053117 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.053095    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-gl87v_odh-ai-gateway-infra(8e5f06ab-42af-4d39-9776-f4f7d3009c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c"
Oct 30 12:30:06.054673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.054655    4453 scope.go:122] "RemoveContainer" containerID="c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6"
Oct 30 12:30:06.054933 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.054914    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244_odh-ai-gateway-infra(3aed1b94-6670-47f0-92eb-5d340291567f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" podUID="3aed1b94-6670-47f0-92eb-5d340291567f"
Oct 30 12:30:06.056346 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.056317    4453 scope.go:122] "RemoveContainer" containerID="85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05"
Oct 30 12:30:06.056625 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.056604    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc_odh-ai-gateway-infra(051a1c38-ecee-484c-bf0a-52dc4783379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a"
Oct 30 12:30:06.058100 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.058084    4453 scope.go:122] "RemoveContainer" containerID="0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"
Oct 30 12:30:06.058400 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.058374    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9_odh-ai-gateway-infra(e0b3d23f-5af0-418b-9542-6d555273f0a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5"
Oct 30 12:30:06.059937 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.059919    4453 scope.go:122] "RemoveContainer" containerID="7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42"
Oct 30 12:30:06.060199 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.060178    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279_odh-ai-gateway-infra(52ed9bc8-5a69-44d2-87e9-6f3b15be19d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9"
Oct 30 12:30:06.061716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.061696    4453 scope.go:122] "RemoveContainer" containerID="c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"
Oct 30 12:30:06.061989 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.061965    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk_odh-ai-gateway-infra(1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a"
Oct 30 12:30:06.063229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.063212    4453 scope.go:122] "RemoveContainer" containerID="22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"
Oct 30 12:30:06.063434 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:06.063416    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g_odh-ai-gateway-infra(e0cfa747-2078-4af2-97c2-0d3e02a8e8ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed"
Oct 30 12:30:06.067282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.067265    4453 scope.go:122] "RemoveContainer" containerID="b9a8dbdda8a2992c759e65731d9f62dcf204a90c26ddf405aedb48398bb0c5af"
Oct 30 12:30:06.342655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:06.342576    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:30:07.178361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:07.177317    4453 scope.go:122] "RemoveContainer" containerID="a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"
Oct 30 12:30:07.178361 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:07.177888    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl_odh-ai-gateway-infra(999feecb-154b-4d75-85fd-08ea8fc965e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8"
Oct 30 12:30:07.196476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:07.194416    4453 scope.go:122] "RemoveContainer" containerID="f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"
Oct 30 12:30:07.196476 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:07.194827    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-gl87v_odh-ai-gateway-infra(8e5f06ab-42af-4d39-9776-f4f7d3009c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c"
Oct 30 12:30:07.573814 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:07.573736    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:30:11.343926 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:11.343875    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing" containerID="cri-o://961fe6606bb4d7467ad23b1010be8db88488bca05cd936473f8ccfc6b2b23ec9" gracePeriod=25
Oct 30 12:30:12.625836 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:12.625794    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:30:15.907584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:15.907210    4453 scope.go:122] "RemoveContainer" containerID="6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11"
Oct 30 12:30:15.908149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:15.908125    4453 scope.go:122] "RemoveContainer" containerID="85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05"
Oct 30 12:30:16.905915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:16.905008    4453 scope.go:122] "RemoveContainer" containerID="f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"
Oct 30 12:30:16.906114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:16.905926    4453 scope.go:122] "RemoveContainer" containerID="0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"
Oct 30 12:30:17.343742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.343705    4453 generic.go:356] "Generic (PLEG): container finished" podID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerID="e618fdbea892619bd9ee8fb449daacfdabc899903d009090a5ee84c72895f01d" exitCode=60
Oct 30 12:30:17.344196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.343768    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerDied","Data":"e618fdbea892619bd9ee8fb449daacfdabc899903d009090a5ee84c72895f01d"}
Oct 30 12:30:17.344196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.343832    4453 scope.go:122] "RemoveContainer" containerID="6caa0fea3ca25b5c445a0a14bc0fc911ef7378867733a9bfd1cf6f8090028e11"
Oct 30 12:30:17.344276 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.344264    4453 scope.go:122] "RemoveContainer" containerID="e618fdbea892619bd9ee8fb449daacfdabc899903d009090a5ee84c72895f01d"
Oct 30 12:30:17.344702 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:17.344676    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg_odh-ai-gateway-infra(0c5dad6e-ec23-4f28-a550-cd49b2f7bc05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05"
Oct 30 12:30:17.346909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.346837    4453 generic.go:356] "Generic (PLEG): container finished" podID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerID="9656d6e5ae7b71795a214db50823f33a79bde250dd4537859399926ccb640073" exitCode=60
Oct 30 12:30:17.346997 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.346945    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" event={"ID":"051a1c38-ecee-484c-bf0a-52dc4783379a","Type":"ContainerDied","Data":"9656d6e5ae7b71795a214db50823f33a79bde250dd4537859399926ccb640073"}
Oct 30 12:30:17.347480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.347459    4453 scope.go:122] "RemoveContainer" containerID="9656d6e5ae7b71795a214db50823f33a79bde250dd4537859399926ccb640073"
Oct 30 12:30:17.347905 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:17.347878    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc_odh-ai-gateway-infra(051a1c38-ecee-484c-bf0a-52dc4783379a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a"
Oct 30 12:30:17.359137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.359114    4453 scope.go:122] "RemoveContainer" containerID="85400240b4dbf91fdf868b393bcb1a77caca355baaeec09fef73832d675acd05"
Oct 30 12:30:17.905129 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.905099    4453 scope.go:122] "RemoveContainer" containerID="c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6"
Oct 30 12:30:17.905303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:17.905259    4453 scope.go:122] "RemoveContainer" containerID="c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"
Oct 30 12:30:18.358602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.358568    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerID="8b29697e40574a3f40f8c1ff7bc4d85bde5bf74976aae685a742df6636dbbb29" exitCode=60
Oct 30 12:30:18.359043 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.358637    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerDied","Data":"8b29697e40574a3f40f8c1ff7bc4d85bde5bf74976aae685a742df6636dbbb29"}
Oct 30 12:30:18.359043 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.358697    4453 scope.go:122] "RemoveContainer" containerID="f54c3493e667df822d0799faf8c8f6d24f97f4b7e2a687a69caa19290be18066"
Oct 30 12:30:18.359233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.359214    4453 scope.go:122] "RemoveContainer" containerID="8b29697e40574a3f40f8c1ff7bc4d85bde5bf74976aae685a742df6636dbbb29"
Oct 30 12:30:18.359683 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:18.359655    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889390-gl87v_odh-ai-gateway-infra(8e5f06ab-42af-4d39-9776-f4f7d3009c2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c"
Oct 30 12:30:18.362294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.362272    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerStarted","Data":"69b9da39069e2618cf02fef9e2768feb108e9f9d1d0174fa5e057e3a06ead8e1"}
Oct 30 12:30:18.391763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.391731    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:18.427168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.427140    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:18.604313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.604280    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:18.690582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.689455    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:18.699318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.699285    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\" (UniqueName: \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\") pod \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") "
Oct 30 12:30:18.699526 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.699507    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\") pod \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\" (UID: \"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05\") "
Oct 30 12:30:18.699913 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.699887    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca" pod "0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" (UID: "0c5dad6e-ec23-4f28-a550-cd49b2f7bc05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:18.700361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.700309    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:18.701795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.701774    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74" pod "0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" (UID: "0c5dad6e-ec23-4f28-a550-cd49b2f7bc05"). InnerVolumeSpecName "kube-api-access-nhb74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:18.801294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.801259    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\") pod \"051a1c38-ecee-484c-bf0a-52dc4783379a\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") "
Oct 30 12:30:18.801523 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.801498    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\" (UniqueName: \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\") pod \"051a1c38-ecee-484c-bf0a-52dc4783379a\" (UID: \"051a1c38-ecee-484c-bf0a-52dc4783379a\") "
Oct 30 12:30:18.801938 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.801689    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca" pod "051a1c38-ecee-484c-bf0a-52dc4783379a" (UID: "051a1c38-ecee-484c-bf0a-52dc4783379a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:18.802629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.802597    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051a1c38-ecee-484c-bf0a-52dc4783379a-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:18.802730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.802629    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhb74\" (UniqueName: \"kubernetes.io/projected/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05-kube-api-access-nhb74\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:18.803958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.803932    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64" pod "051a1c38-ecee-484c-bf0a-52dc4783379a" (UID: "051a1c38-ecee-484c-bf0a-52dc4783379a"). InnerVolumeSpecName "kube-api-access-skc64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:18.903868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.903832    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skc64\" (UniqueName: \"kubernetes.io/projected/051a1c38-ecee-484c-bf0a-52dc4783379a-kube-api-access-skc64\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:18.904660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.904641    4453 scope.go:122] "RemoveContainer" containerID="22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"
Oct 30 12:30:18.904781 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:18.904759    4453 scope.go:122] "RemoveContainer" containerID="a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"
Oct 30 12:30:19.371926 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.371899    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"
Oct 30 12:30:19.372372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.371897    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc" event={"ID":"051a1c38-ecee-484c-bf0a-52dc4783379a","Type":"ContainerDied","Data":"537d86dc8db898b8a83f5e2dbbcdc01e04e351f0ff36c48a89aa1365a3d4ac98"}
Oct 30 12:30:19.372372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.372034    4453 scope.go:122] "RemoveContainer" containerID="9656d6e5ae7b71795a214db50823f33a79bde250dd4537859399926ccb640073"
Oct 30 12:30:19.374127 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.374101    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerID="69b9da39069e2618cf02fef9e2768feb108e9f9d1d0174fa5e057e3a06ead8e1" exitCode=60
Oct 30 12:30:19.374278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.374158    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerDied","Data":"69b9da39069e2618cf02fef9e2768feb108e9f9d1d0174fa5e057e3a06ead8e1"}
Oct 30 12:30:19.374705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.374683    4453 scope.go:122] "RemoveContainer" containerID="69b9da39069e2618cf02fef9e2768feb108e9f9d1d0174fa5e057e3a06ead8e1"
Oct 30 12:30:19.375131 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:19.375106    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9_odh-ai-gateway-infra(e0b3d23f-5af0-418b-9542-6d555273f0a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5"
Oct 30 12:30:19.376611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.376586    4453 generic.go:356] "Generic (PLEG): container finished" podID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerID="0516751ae1fe6ba515a7f1864d60b79a9c01015bff4f7e28778cb8648e78f970" exitCode=60
Oct 30 12:30:19.376719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.376618    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerDied","Data":"0516751ae1fe6ba515a7f1864d60b79a9c01015bff4f7e28778cb8648e78f970"}
Oct 30 12:30:19.377111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.377090    4453 scope.go:122] "RemoveContainer" containerID="0516751ae1fe6ba515a7f1864d60b79a9c01015bff4f7e28778cb8648e78f970"
Oct 30 12:30:19.377529 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:19.377508    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk_odh-ai-gateway-infra(1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a"
Oct 30 12:30:19.378721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.378668    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg" event={"ID":"0c5dad6e-ec23-4f28-a550-cd49b2f7bc05","Type":"ContainerDied","Data":"60c9816fa06a3e77e19d415d696dc59c8b88d4b3b60ad00a6eff0952d4c44e03"}
Oct 30 12:30:19.378952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.378939    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"
Oct 30 12:30:19.427244 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.427204    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:19.447425 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.447340    4453 generic.go:356] "Generic (PLEG): container finished" podID="3aed1b94-6670-47f0-92eb-5d340291567f" containerID="ce1de354ce6ac79abf4e51521c755cc10a4bc3eb0d16a2d36a8f31c956bf3a62" exitCode=60
Oct 30 12:30:19.447425 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.447408    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" event={"ID":"3aed1b94-6670-47f0-92eb-5d340291567f","Type":"ContainerDied","Data":"ce1de354ce6ac79abf4e51521c755cc10a4bc3eb0d16a2d36a8f31c956bf3a62"}
Oct 30 12:30:19.448027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.448003    4453 scope.go:122] "RemoveContainer" containerID="ce1de354ce6ac79abf4e51521c755cc10a4bc3eb0d16a2d36a8f31c956bf3a62"
Oct 30 12:30:19.449746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.449724    4453 scope.go:122] "RemoveContainer" containerID="0abb4ab5d2c62328fe5aa06bfc2ca4c7b6901712819cea132e08866e5daa6c25"
Oct 30 12:30:19.449746 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:19.449726    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244_odh-ai-gateway-infra(3aed1b94-6670-47f0-92eb-5d340291567f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" podUID="3aed1b94-6670-47f0-92eb-5d340291567f"
Oct 30 12:30:19.450020 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.449993    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:19.469759 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.469734    4453 scope.go:122] "RemoveContainer" containerID="c47baf3c21f4e64720c2fc642fda8df38cbc72ffdb3c059a0a552d90e3517942"
Oct 30 12:30:19.484460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.484433    4453 scope.go:122] "RemoveContainer" containerID="e618fdbea892619bd9ee8fb449daacfdabc899903d009090a5ee84c72895f01d"
Oct 30 12:30:19.497068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.497047    4453 scope.go:122] "RemoveContainer" containerID="c6f2e0791e492999e1388ab626979742715177888a2d6b371b2dde7613889aa6"
Oct 30 12:30:19.560803 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.560769    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:19.567538 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.567506    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-5cf206-29889390-52khc"]
Oct 30 12:30:19.599812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.599785    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:19.609609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.609574    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-59b8a9-29889390-2qtwg"]
Oct 30 12:30:19.642846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.642817    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:19.822994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.822905    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\") pod \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") "
Oct 30 12:30:19.823161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.823128    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\" (UniqueName: \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\") pod \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\" (UID: \"8e5f06ab-42af-4d39-9776-f4f7d3009c2c\") "
Oct 30 12:30:19.823367 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.823324    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca" pod "8e5f06ab-42af-4d39-9776-f4f7d3009c2c" (UID: "8e5f06ab-42af-4d39-9776-f4f7d3009c2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:19.823681 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.823663    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:19.825407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.825285    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs" pod "8e5f06ab-42af-4d39-9776-f4f7d3009c2c" (UID: "8e5f06ab-42af-4d39-9776-f4f7d3009c2c"). InnerVolumeSpecName "kube-api-access-phhrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:19.905500 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.905464    4453 scope.go:122] "RemoveContainer" containerID="7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42"
Oct 30 12:30:19.925701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:19.925665    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhrs\" (UniqueName: \"kubernetes.io/projected/8e5f06ab-42af-4d39-9776-f4f7d3009c2c-kube-api-access-phhrs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:20.458477 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.458437    4453 generic.go:356] "Generic (PLEG): container finished" podID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerID="738009abe37787f3920e3db62f3a6ced4411c9236f973ec8f0e6f5ff43cfae46" exitCode=60
Oct 30 12:30:20.458951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.458510    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerDied","Data":"738009abe37787f3920e3db62f3a6ced4411c9236f973ec8f0e6f5ff43cfae46"}
Oct 30 12:30:20.458951 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.458581    4453 scope.go:122] "RemoveContainer" containerID="a04843bb23fed071f45fcc97c7f085b129b86b7f7a21b576a6f01c825b7f249c"
Oct 30 12:30:20.459047 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.459009    4453 scope.go:122] "RemoveContainer" containerID="738009abe37787f3920e3db62f3a6ced4411c9236f973ec8f0e6f5ff43cfae46"
Oct 30 12:30:20.459483 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:20.459457    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl_odh-ai-gateway-infra(999feecb-154b-4d75-85fd-08ea8fc965e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8"
Oct 30 12:30:20.460123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.460102    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"
Oct 30 12:30:20.460242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.460103    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v" event={"ID":"8e5f06ab-42af-4d39-9776-f4f7d3009c2c","Type":"ContainerDied","Data":"366ccdc737d26429acd4d6bbcedeb154648345c0758131226c85964ba518fd8f"}
Oct 30 12:30:20.468237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.468203    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerStarted","Data":"51ed09a0a4cf8dc0ee2eb9dd111575a42024ff215216e18b37b36fecc6ec4411"}
Oct 30 12:30:20.472628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.471592    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:20.473653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.473618    4453 generic.go:356] "Generic (PLEG): container finished" podID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerID="92a738983efbcc27af82056b1948fb3e8932045daa88149b94becc0bcbbdd049" exitCode=60
Oct 30 12:30:20.473755 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.473674    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerDied","Data":"92a738983efbcc27af82056b1948fb3e8932045daa88149b94becc0bcbbdd049"}
Oct 30 12:30:20.474315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.474256    4453 scope.go:122] "RemoveContainer" containerID="92a738983efbcc27af82056b1948fb3e8932045daa88149b94becc0bcbbdd049"
Oct 30 12:30:20.474743 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:20.474717    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g_odh-ai-gateway-infra(e0cfa747-2078-4af2-97c2-0d3e02a8e8ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed"
Oct 30 12:30:20.478364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.478345    4453 scope.go:122] "RemoveContainer" containerID="8b29697e40574a3f40f8c1ff7bc4d85bde5bf74976aae685a742df6636dbbb29"
Oct 30 12:30:20.491650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.491604    4453 scope.go:122] "RemoveContainer" containerID="22a17795dd5118802fb18aa77d8c99dff980434d28f199ff230b12ef0e7a3001"
Oct 30 12:30:20.569190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.569131    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" podStartSLOduration=20.56911755 podStartE2EDuration="20.56911755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:30:20.565619597 +0000 UTC m=+3619.785904504" watchObservedRunningTime="2026-10-30 12:30:20.56911755 +0000 UTC m=+3619.789402462"
Oct 30 12:30:20.650629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.650512    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:20.706842 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.688888    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:20.710668 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.710524    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889390-gl87v"]
Oct 30 12:30:20.758292 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.758258    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:20.825268 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.825235    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:20.881758 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.881732    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:20.917836 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.917748    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" path="/var/lib/kubelet/pods/051a1c38-ecee-484c-bf0a-52dc4783379a/volumes"
Oct 30 12:30:20.918549 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.918521    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" path="/var/lib/kubelet/pods/0c5dad6e-ec23-4f28-a550-cd49b2f7bc05/volumes"
Oct 30 12:30:20.919163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.919139    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" path="/var/lib/kubelet/pods/8e5f06ab-42af-4d39-9776-f4f7d3009c2c/volumes"
Oct 30 12:30:20.945686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.945656    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\") pod \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") "
Oct 30 12:30:20.945867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.945709    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\" (UniqueName: \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\") pod \"e0b3d23f-5af0-418b-9542-6d555273f0a5\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") "
Oct 30 12:30:20.945867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.945825    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\") pod \"e0b3d23f-5af0-418b-9542-6d555273f0a5\" (UID: \"e0b3d23f-5af0-418b-9542-6d555273f0a5\") "
Oct 30 12:30:20.946032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.945875    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\") pod \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\" (UID: \"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a\") "
Oct 30 12:30:20.946219 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.946195    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca" pod "e0b3d23f-5af0-418b-9542-6d555273f0a5" (UID: "e0b3d23f-5af0-418b-9542-6d555273f0a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:20.946383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.946314    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca" pod "1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" (UID: "1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:20.946582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.946563    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b3d23f-5af0-418b-9542-6d555273f0a5-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:20.946638 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.946589    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:20.948162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.948136    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4" pod "e0b3d23f-5af0-418b-9542-6d555273f0a5" (UID: "e0b3d23f-5af0-418b-9542-6d555273f0a5"). InnerVolumeSpecName "kube-api-access-kpck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:20.949789 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:20.949764    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c" pod "1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" (UID: "1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a"). InnerVolumeSpecName "kube-api-access-tr75c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:21.058444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.056531    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\" (UniqueName: \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\") pod \"3aed1b94-6670-47f0-92eb-5d340291567f\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") "
Oct 30 12:30:21.058444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.056617    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\") pod \"3aed1b94-6670-47f0-92eb-5d340291567f\" (UID: \"3aed1b94-6670-47f0-92eb-5d340291567f\") "
Oct 30 12:30:21.063915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.063873    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca" pod "3aed1b94-6670-47f0-92eb-5d340291567f" (UID: "3aed1b94-6670-47f0-92eb-5d340291567f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:21.077971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.072402    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aed1b94-6670-47f0-92eb-5d340291567f-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.077971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.072438    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr75c\" (UniqueName: \"kubernetes.io/projected/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a-kube-api-access-tr75c\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.077971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.072453    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpck4\" (UniqueName: \"kubernetes.io/projected/e0b3d23f-5af0-418b-9542-6d555273f0a5-kube-api-access-kpck4\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.077971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.077916    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk" pod "3aed1b94-6670-47f0-92eb-5d340291567f" (UID: "3aed1b94-6670-47f0-92eb-5d340291567f"). InnerVolumeSpecName "kube-api-access-lhbdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:21.174709 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.174623    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbdk\" (UniqueName: \"kubernetes.io/projected/3aed1b94-6670-47f0-92eb-5d340291567f-kube-api-access-lhbdk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.483861 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.483784    4453 generic.go:356] "Generic (PLEG): container finished" podID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerID="51ed09a0a4cf8dc0ee2eb9dd111575a42024ff215216e18b37b36fecc6ec4411" exitCode=60
Oct 30 12:30:21.484270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.483855    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerDied","Data":"51ed09a0a4cf8dc0ee2eb9dd111575a42024ff215216e18b37b36fecc6ec4411"}
Oct 30 12:30:21.484270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.483937    4453 scope.go:122] "RemoveContainer" containerID="7141ac9dd31d6f9921478d2b260cd794685b8cfdd138940ee500cf8b566bee42"
Oct 30 12:30:21.484418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.484406    4453 scope.go:122] "RemoveContainer" containerID="51ed09a0a4cf8dc0ee2eb9dd111575a42024ff215216e18b37b36fecc6ec4411"
Oct 30 12:30:21.484804 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:21.484780    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279_odh-ai-gateway-infra(52ed9bc8-5a69-44d2-87e9-6f3b15be19d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9"
Oct 30 12:30:21.485775 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.485756    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"
Oct 30 12:30:21.485867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.485794    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk" event={"ID":"1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a","Type":"ContainerDied","Data":"f1b90d692fd761b1e45cbd2f6941030bf95ee4d705b94b84c0722365efb8514a"}
Oct 30 12:30:21.492128 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.492106    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"
Oct 30 12:30:21.492245 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.492145    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244" event={"ID":"3aed1b94-6670-47f0-92eb-5d340291567f","Type":"ContainerDied","Data":"7157106cf22ebe74cf91b673b87b72f3799383fc0134e6a5c54e87548ac9595e"}
Oct 30 12:30:21.494393 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.494366    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9" event={"ID":"e0b3d23f-5af0-418b-9542-6d555273f0a5","Type":"ContainerDied","Data":"286bf3080d13881a3e820a76bf3fea9da3364a6aa422098e5c09b0fc7e0e96b8"}
Oct 30 12:30:21.494393 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.494389    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"
Oct 30 12:30:21.501244 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.501224    4453 scope.go:122] "RemoveContainer" containerID="0516751ae1fe6ba515a7f1864d60b79a9c01015bff4f7e28778cb8648e78f970"
Oct 30 12:30:21.510106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.510080    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:21.513696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.513673    4453 scope.go:122] "RemoveContainer" containerID="ce1de354ce6ac79abf4e51521c755cc10a4bc3eb0d16a2d36a8f31c956bf3a62"
Oct 30 12:30:21.527074 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.527031    4453 scope.go:122] "RemoveContainer" containerID="69b9da39069e2618cf02fef9e2768feb108e9f9d1d0174fa5e057e3a06ead8e1"
Oct 30 12:30:21.579765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.579724    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:21.609373 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.602303    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:21.609785 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.609759    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-31a6e3-29889390-v9bpk"]
Oct 30 12:30:21.634826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.634777    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:21.641250 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.641214    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9b2407-29889390-m65x9"]
Oct 30 12:30:21.653562 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.653531    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:21.676121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.676091    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:21.682222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.682194    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b02580-29889390-26244"]
Oct 30 12:30:21.710520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.710491    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:21.789028 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.788947    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\" (UniqueName: \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\") pod \"999feecb-154b-4d75-85fd-08ea8fc965e8\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") "
Oct 30 12:30:21.789201 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.789153    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\") pod \"999feecb-154b-4d75-85fd-08ea8fc965e8\" (UID: \"999feecb-154b-4d75-85fd-08ea8fc965e8\") "
Oct 30 12:30:21.789610 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.789577    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca" pod "999feecb-154b-4d75-85fd-08ea8fc965e8" (UID: "999feecb-154b-4d75-85fd-08ea8fc965e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:21.790095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.790067    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/999feecb-154b-4d75-85fd-08ea8fc965e8-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.791448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.791420    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9" pod "999feecb-154b-4d75-85fd-08ea8fc965e8" (UID: "999feecb-154b-4d75-85fd-08ea8fc965e8"). InnerVolumeSpecName "kube-api-access-gq4m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:21.834862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.834834    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:21.891281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.891241    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\" (UniqueName: \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\") pod \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") "
Oct 30 12:30:21.891496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.891318    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\") pod \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\" (UID: \"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed\") "
Oct 30 12:30:21.892020 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.891990    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca" pod "e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" (UID: "e0cfa747-2078-4af2-97c2-0d3e02a8e8ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:21.892144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.892098    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4m9\" (UniqueName: \"kubernetes.io/projected/999feecb-154b-4d75-85fd-08ea8fc965e8-kube-api-access-gq4m9\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.893712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.893684    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99" pod "e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" (UID: "e0cfa747-2078-4af2-97c2-0d3e02a8e8ed"). InnerVolumeSpecName "kube-api-access-sqh99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:21.994318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.994282    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqh99\" (UniqueName: \"kubernetes.io/projected/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-kube-api-access-sqh99\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:21.994318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:21.994314    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.503547 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.503505    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"
Oct 30 12:30:22.503963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.503545    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g" event={"ID":"e0cfa747-2078-4af2-97c2-0d3e02a8e8ed","Type":"ContainerDied","Data":"fd8d53c8e60c8b3644364fe5e7f68d96cea4f90490502482cb20d1540c73b487"}
Oct 30 12:30:22.503963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.503617    4453 scope.go:122] "RemoveContainer" containerID="92a738983efbcc27af82056b1948fb3e8932045daa88149b94becc0bcbbdd049"
Oct 30 12:30:22.505203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.505178    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"
Oct 30 12:30:22.505345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.505204    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl" event={"ID":"999feecb-154b-4d75-85fd-08ea8fc965e8","Type":"ContainerDied","Data":"d3124f45f216469648c299be74b319d3cc7789ceb1ca8eb8575b7a267b8d8fd8"}
Oct 30 12:30:22.516142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.516121    4453 scope.go:122] "RemoveContainer" containerID="738009abe37787f3920e3db62f3a6ced4411c9236f973ec8f0e6f5ff43cfae46"
Oct 30 12:30:22.604982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.601066    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:22.609911 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.609877    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ffea51-29889390-lrb5g"]
Oct 30 12:30:22.626537 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.626500    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:30:22.633757 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.633704    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:22.642234 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.642198    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e43a66-29889390-7s9xl"]
Oct 30 12:30:22.696556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.696532    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:22.808361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.808304    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\") pod \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") "
Oct 30 12:30:22.808703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.808684    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\" (UniqueName: \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\") pod \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\" (UID: \"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9\") "
Oct 30 12:30:22.808759 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.808733    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca" pod "52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" (UID: "52ed9bc8-5a69-44d2-87e9-6f3b15be19d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:22.809198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.809180    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.810983 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.810960    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4" pod "52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" (UID: "52ed9bc8-5a69-44d2-87e9-6f3b15be19d9"). InnerVolumeSpecName "kube-api-access-ld6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:22.910631 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.910600    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld6f4\" (UniqueName: \"kubernetes.io/projected/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9-kube-api-access-ld6f4\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:22.915618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.915594    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" path="/var/lib/kubelet/pods/1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a/volumes"
Oct 30 12:30:22.916044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.916029    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" path="/var/lib/kubelet/pods/3aed1b94-6670-47f0-92eb-5d340291567f/volumes"
Oct 30 12:30:22.916518 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.916502    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" path="/var/lib/kubelet/pods/999feecb-154b-4d75-85fd-08ea8fc965e8/volumes"
Oct 30 12:30:22.917237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.917221    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" path="/var/lib/kubelet/pods/e0b3d23f-5af0-418b-9542-6d555273f0a5/volumes"
Oct 30 12:30:22.917666 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:22.917651    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" path="/var/lib/kubelet/pods/e0cfa747-2078-4af2-97c2-0d3e02a8e8ed/volumes"
Oct 30 12:30:23.523786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:23.523751    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279" event={"ID":"52ed9bc8-5a69-44d2-87e9-6f3b15be19d9","Type":"ContainerDied","Data":"1c63c792e0fdd4bfde7226cfcabb6bb4f1b696bcd6971eb29d30f1bfa14df6aa"}
Oct 30 12:30:23.523786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:23.523776    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"
Oct 30 12:30:23.524285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:23.523805    4453 scope.go:122] "RemoveContainer" containerID="51ed09a0a4cf8dc0ee2eb9dd111575a42024ff215216e18b37b36fecc6ec4411"
Oct 30 12:30:23.573644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:23.573611    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:23.582588 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:23.582561    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d71fa-29889390-9v279"]
Oct 30 12:30:24.915680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:24.915649    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" path="/var/lib/kubelet/pods/52ed9bc8-5a69-44d2-87e9-6f3b15be19d9/volumes"
Oct 30 12:30:31.605431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.605392    4453 generic.go:356] "Generic (PLEG): container finished" podID="3c911504-d363-4f1f-a526-3f98f4810061" containerID="961fe6606bb4d7467ad23b1010be8db88488bca05cd936473f8ccfc6b2b23ec9" exitCode=0
Oct 30 12:30:31.605845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.605491    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" event={"ID":"3c911504-d363-4f1f-a526-3f98f4810061","Type":"ContainerDied","Data":"961fe6606bb4d7467ad23b1010be8db88488bca05cd936473f8ccfc6b2b23ec9"}
Oct 30 12:30:31.605845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.605561    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg" event={"ID":"3c911504-d363-4f1f-a526-3f98f4810061","Type":"ContainerDied","Data":"39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817"}
Oct 30 12:30:31.605845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.605586    4453 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b7758b1ea4ddace915d80f2cf368002f60a40619af7f0ba7abacd867a44817"
Oct 30 12:30:31.631054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.631033    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:30:31.760847 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.760817    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\") pod \"3c911504-d363-4f1f-a526-3f98f4810061\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") "
Oct 30 12:30:31.761027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.761013    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\" (UniqueName: \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\") pod \"3c911504-d363-4f1f-a526-3f98f4810061\" (UID: \"3c911504-d363-4f1f-a526-3f98f4810061\") "
Oct 30 12:30:31.761222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.761200    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume" pod "3c911504-d363-4f1f-a526-3f98f4810061" (UID: "3c911504-d363-4f1f-a526-3f98f4810061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:31.761611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.761587    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c911504-d363-4f1f-a526-3f98f4810061-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:31.763165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.763143    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg" pod "3c911504-d363-4f1f-a526-3f98f4810061" (UID: "3c911504-d363-4f1f-a526-3f98f4810061"). InnerVolumeSpecName "kube-api-access-vlbmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:31.863361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:31.863318    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlbmg\" (UniqueName: \"kubernetes.io/projected/3c911504-d363-4f1f-a526-3f98f4810061-kube-api-access-vlbmg\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:32.612956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:32.612915    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"
Oct 30 12:30:32.685484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:32.685445    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:30:32.695231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:32.695196    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a1f322-78f78948cf-w8gcg"]
Oct 30 12:30:32.913480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:32.913399    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c911504-d363-4f1f-a526-3f98f4810061" path="/var/lib/kubelet/pods/3c911504-d363-4f1f-a526-3f98f4810061/volumes"
Oct 30 12:30:37.576008 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:37.575967    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:30:45.369983 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:45.369888    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:30:45.370431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:45.370149    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy" containerID="cri-o://dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc" gracePeriod=30
Oct 30 12:30:50.674325 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.674297    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735644    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735798    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735848    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735893    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735914    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735930    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.735985    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.736053    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.736968 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.736114    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\") pod \"ffea7c47-cd3e-4738-928c-3a680d656825\" (UID: \"ffea7c47-cd3e-4738-928c-3a680d656825\") "
Oct 30 12:30:50.737491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.737427    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.737980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.737938    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:30:50.738342 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.738300    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.738591 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.738540    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.743152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.738897    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.743152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.740735    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "kube-api-access-slj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:50.743152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.740995    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:30:50.743152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.741271    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:30:50.747793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.747766    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token" pod "ffea7c47-cd3e-4738-928c-3a680d656825" (UID: "ffea7c47-cd3e-4738-928c-3a680d656825"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876670    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876703    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffea7c47-cd3e-4738-928c-3a680d656825-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876718    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876732    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slj6h\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-kube-api-access-slj6h\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876747    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876760    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876773    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffea7c47-cd3e-4738-928c-3a680d656825-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876787    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffea7c47-cd3e-4738-928c-3a680d656825-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.877476 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.876799    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffea7c47-cd3e-4738-928c-3a680d656825-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:30:50.886912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.882784    4453 generic.go:356] "Generic (PLEG): container finished" podID="ffea7c47-cd3e-4738-928c-3a680d656825" containerID="dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc" exitCode=0
Oct 30 12:30:50.886912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.882910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" event={"ID":"ffea7c47-cd3e-4738-928c-3a680d656825","Type":"ContainerDied","Data":"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"}
Oct 30 12:30:50.886912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.882965    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm" event={"ID":"ffea7c47-cd3e-4738-928c-3a680d656825","Type":"ContainerDied","Data":"7c613e36d1c5cb3c55f023801d3b2e720e0ee9cc1922059ec86b8035b087eb30"}
Oct 30 12:30:50.886912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.882997    4453 scope.go:122] "RemoveContainer" containerID="dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"
Oct 30 12:30:50.886912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.883168    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"
Oct 30 12:30:50.932415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.932363    4453 scope.go:122] "RemoveContainer" containerID="dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"
Oct 30 12:30:50.932761 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:30:50.932731    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc\": container with ID starting with dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc not found: ID does not exist" containerID="dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"
Oct 30 12:30:50.932829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.932777    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc"} err="failed to get container status \"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc\": rpc error: code = NotFound desc = could not find container \"dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc\": container with ID starting with dac82ff5d6d16a0cebd2580595e93dd047f2baeb99912331b45140affbafc4cc not found: ID does not exist"
Oct 30 12:30:50.965607 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.965576    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:30:50.971861 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:50.971823    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a1f322-openshift-default-64b5c455c-xslvm"]
Oct 30 12:30:52.913998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:52.913960    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" path="/var/lib/kubelet/pods/ffea7c47-cd3e-4738-928c-3a680d656825/volumes"
Oct 30 12:30:56.482721 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:56.482676    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:30:56.513663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:30:56.513626    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:31:01.484705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:01.484665    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing" containerID="cri-o://58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75" gracePeriod=25
Oct 30 12:31:01.514973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:01.514918    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing" containerID="cri-o://bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6" gracePeriod=25
Oct 30 12:31:01.562383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:01.562345    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:31:02.099980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:02.099946    4453 scope.go:122] "RemoveContainer" containerID="961fe6606bb4d7467ad23b1010be8db88488bca05cd936473f8ccfc6b2b23ec9"
Oct 30 12:31:04.400611 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:04.400571    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:05.408380 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:05.408311    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:06.562881 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:06.562840    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing" containerID="cri-o://349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618" gracePeriod=25
Oct 30 12:31:07.572791 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:07.572757    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:31:11.780535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:11.780492    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:14.400508 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:14.400468    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:15.408265 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:15.408223    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:21.781812 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.781408    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:21.801002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.800967    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:31:21.821644 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.821616    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:31:21.858933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.858899    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:31:21.981980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.981946    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\") pod \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") "
Oct 30 12:31:21.982165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.982041    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\" (UniqueName: \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\") pod \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\" (UID: \"8e50b525-461a-40a4-aa8e-e5bafaa95f6e\") "
Oct 30 12:31:21.982376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.982353    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\") pod \"20717866-3b47-441f-8deb-e1f810cb9c43\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") "
Oct 30 12:31:21.982483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.982360    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume" pod "8e50b525-461a-40a4-aa8e-e5bafaa95f6e" (UID: "8e50b525-461a-40a4-aa8e-e5bafaa95f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:21.982623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.982518    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\" (UniqueName: \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\") pod \"20717866-3b47-441f-8deb-e1f810cb9c43\" (UID: \"20717866-3b47-441f-8deb-e1f810cb9c43\") "
Oct 30 12:31:21.982703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.982674    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume" pod "20717866-3b47-441f-8deb-e1f810cb9c43" (UID: "20717866-3b47-441f-8deb-e1f810cb9c43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:21.983021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.983002    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:21.983021 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.983021    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20717866-3b47-441f-8deb-e1f810cb9c43-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:21.984152 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.984134    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh" pod "8e50b525-461a-40a4-aa8e-e5bafaa95f6e" (UID: "8e50b525-461a-40a4-aa8e-e5bafaa95f6e"). InnerVolumeSpecName "kube-api-access-7mhgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:21.984472 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:21.984455    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c" pod "20717866-3b47-441f-8deb-e1f810cb9c43" (UID: "20717866-3b47-441f-8deb-e1f810cb9c43"). InnerVolumeSpecName "kube-api-access-8zn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:22.085070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.084996    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zn9c\" (UniqueName: \"kubernetes.io/projected/20717866-3b47-441f-8deb-e1f810cb9c43-kube-api-access-8zn9c\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:22.085070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.085023    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhgh\" (UniqueName: \"kubernetes.io/projected/8e50b525-461a-40a4-aa8e-e5bafaa95f6e-kube-api-access-7mhgh\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:22.235461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.235421    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerID="58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75" exitCode=0
Oct 30 12:31:22.235680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.235665    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"
Oct 30 12:31:22.236299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.236271    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" event={"ID":"8e50b525-461a-40a4-aa8e-e5bafaa95f6e","Type":"ContainerDied","Data":"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"}
Oct 30 12:31:22.236512 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.236491    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj" event={"ID":"8e50b525-461a-40a4-aa8e-e5bafaa95f6e","Type":"ContainerDied","Data":"d5e347e21bd1dda4eabd53109d20ce1601ff25e99a227df02f58a11f75fe7ea3"}
Oct 30 12:31:22.236649 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.236623    4453 scope.go:122] "RemoveContainer" containerID="58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"
Oct 30 12:31:22.245506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.243621    4453 generic.go:356] "Generic (PLEG): container finished" podID="20717866-3b47-441f-8deb-e1f810cb9c43" containerID="bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6" exitCode=0
Oct 30 12:31:22.245506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.243719    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" event={"ID":"20717866-3b47-441f-8deb-e1f810cb9c43","Type":"ContainerDied","Data":"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"}
Oct 30 12:31:22.245506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.243757    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh" event={"ID":"20717866-3b47-441f-8deb-e1f810cb9c43","Type":"ContainerDied","Data":"bb28fefe89b61f30eebcb991d823fc3454da8fb1e15143b83c1889518b0f572d"}
Oct 30 12:31:22.245506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.243825    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"
Oct 30 12:31:22.256429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.256408    4453 scope.go:122] "RemoveContainer" containerID="58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"
Oct 30 12:31:22.256812 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:22.256786    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75\": container with ID starting with 58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75 not found: ID does not exist" containerID="58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"
Oct 30 12:31:22.256889 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.256823    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75"} err="failed to get container status \"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75\": rpc error: code = NotFound desc = could not find container \"58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75\": container with ID starting with 58f79927ca89704c14c70c47b7f793d2c8e21ddfddfea0624ab7588bf54f0f75 not found: ID does not exist"
Oct 30 12:31:22.256889 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.256845    4453 scope.go:122] "RemoveContainer" containerID="bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"
Oct 30 12:31:22.268747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.268728    4453 scope.go:122] "RemoveContainer" containerID="bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"
Oct 30 12:31:22.268970 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:22.268951    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6\": container with ID starting with bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6 not found: ID does not exist" containerID="bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"
Oct 30 12:31:22.269011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.268977    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6"} err="failed to get container status \"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6\": rpc error: code = NotFound desc = could not find container \"bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6\": container with ID starting with bf0bfc30ed6227ab97b6c8f34bd11e12661384e4b890af10c3478b6967b372f6 not found: ID does not exist"
Oct 30 12:31:22.298953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.298926    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:31:22.304459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.304434    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b2407-877899c8b-7h2mj"]
Oct 30 12:31:22.330495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.330469    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:31:22.336321 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.336261    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b2407-5b7696dd87-hlfsh"]
Oct 30 12:31:22.914373 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.914311    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" path="/var/lib/kubelet/pods/20717866-3b47-441f-8deb-e1f810cb9c43/volumes"
Oct 30 12:31:22.914813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:22.914797    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" path="/var/lib/kubelet/pods/8e50b525-461a-40a4-aa8e-e5bafaa95f6e/volumes"
Oct 30 12:31:26.802127 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.802070    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing" containerID="cri-o://1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43" gracePeriod=25
Oct 30 12:31:26.853406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.853380    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:31:26.916829 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.916791    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\") pod \"87cda50b-fdb2-4b11-913e-544644360e39\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") "
Oct 30 12:31:26.917160 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.917138    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\" (UniqueName: \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\") pod \"87cda50b-fdb2-4b11-913e-544644360e39\" (UID: \"87cda50b-fdb2-4b11-913e-544644360e39\") "
Oct 30 12:31:26.917278 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.917227    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume" pod "87cda50b-fdb2-4b11-913e-544644360e39" (UID: "87cda50b-fdb2-4b11-913e-544644360e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:26.917831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.917807    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87cda50b-fdb2-4b11-913e-544644360e39-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:26.919738 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:26.919707    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh" pod "87cda50b-fdb2-4b11-913e-544644360e39" (UID: "87cda50b-fdb2-4b11-913e-544644360e39"). InnerVolumeSpecName "kube-api-access-v5gfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:27.019070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.019034    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gfh\" (UniqueName: \"kubernetes.io/projected/87cda50b-fdb2-4b11-913e-544644360e39-kube-api-access-v5gfh\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:27.287869 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.287836    4453 generic.go:356] "Generic (PLEG): container finished" podID="87cda50b-fdb2-4b11-913e-544644360e39" containerID="349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618" exitCode=0
Oct 30 12:31:27.288097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.287903    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"
Oct 30 12:31:27.288097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.287910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" event={"ID":"87cda50b-fdb2-4b11-913e-544644360e39","Type":"ContainerDied","Data":"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"}
Oct 30 12:31:27.288097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.287969    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz" event={"ID":"87cda50b-fdb2-4b11-913e-544644360e39","Type":"ContainerDied","Data":"ee1a79235ad34c5dbc940721fb035f3532d6690b3f9dfc06d380d94bedfbb9df"}
Oct 30 12:31:27.288097 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.287996    4453 scope.go:122] "RemoveContainer" containerID="349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"
Oct 30 12:31:27.300345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.300308    4453 scope.go:122] "RemoveContainer" containerID="349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"
Oct 30 12:31:27.300727 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:27.300701    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618\": container with ID starting with 349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618 not found: ID does not exist" containerID="349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"
Oct 30 12:31:27.300840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.300736    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618"} err="failed to get container status \"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618\": rpc error: code = NotFound desc = could not find container \"349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618\": container with ID starting with 349fa0f44594b41035c116233947243d25074d5ff75333d23417f01ab9e31618 not found: ID does not exist"
Oct 30 12:31:27.334931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.334893    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:31:27.348287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:27.348256    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cf206-65bb5fb8c-ll7sz"]
Oct 30 12:31:28.914091 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:28.914061    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87cda50b-fdb2-4b11-913e-544644360e39" path="/var/lib/kubelet/pods/87cda50b-fdb2-4b11-913e-544644360e39/volumes"
Oct 30 12:31:34.596841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:34.596804    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:37.573767 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:37.573734    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:31:44.596770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:44.596728    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:31:47.124318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.122428    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:31:47.240857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.240826    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\") pod \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") "
Oct 30 12:31:47.241056 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.240905    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\" (UniqueName: \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\") pod \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\" (UID: \"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8\") "
Oct 30 12:31:47.241235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.241210    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume" pod "cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" (UID: "cd7cecb5-1996-4ce0-92f2-21ac8c2339d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:31:47.241474 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.241458    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:47.243091 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.243070    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z" pod "cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" (UID: "cd7cecb5-1996-4ce0-92f2-21ac8c2339d8"). InnerVolumeSpecName "kube-api-access-mqk8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:31:47.342561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.342498    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqk8z\" (UniqueName: \"kubernetes.io/projected/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8-kube-api-access-mqk8z\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:31:47.451374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.450931    4453 generic.go:356] "Generic (PLEG): container finished" podID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerID="1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43" exitCode=0
Oct 30 12:31:47.451374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.451212    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" event={"ID":"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8","Type":"ContainerDied","Data":"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"}
Oct 30 12:31:47.451374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.451267    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7" event={"ID":"cd7cecb5-1996-4ce0-92f2-21ac8c2339d8","Type":"ContainerDied","Data":"91ead7be2930a05a491ab6e39acc7915261daa0275e155334c2424d51700d7a9"}
Oct 30 12:31:47.451374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.451296    4453 scope.go:122] "RemoveContainer" containerID="1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"
Oct 30 12:31:47.451637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.451447    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"
Oct 30 12:31:47.464401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.464382    4453 scope.go:122] "RemoveContainer" containerID="1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"
Oct 30 12:31:47.464691 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:31:47.464672    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43\": container with ID starting with 1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43 not found: ID does not exist" containerID="1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"
Oct 30 12:31:47.464765 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.464697    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43"} err="failed to get container status \"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43\": rpc error: code = NotFound desc = could not find container \"1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43\": container with ID starting with 1bc7a02a4296184d6891115835a0e99fa7dbe2927b7cc781cb6f232be7de4d43 not found: ID does not exist"
Oct 30 12:31:47.497417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.497391    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:31:47.506637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:47.506605    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9d71fa-65b54db46c-krfm7"]
Oct 30 12:31:48.914517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:48.914481    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" path="/var/lib/kubelet/pods/cd7cecb5-1996-4ce0-92f2-21ac8c2339d8/volumes"
Oct 30 12:31:55.161725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:55.161054    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:31:55.167437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:55.167394    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" containerID="cri-o://f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9" gracePeriod=30
Oct 30 12:31:57.815200 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:57.815159    4453 patch_prober.go:29] interesting pod/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:31:57.815620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:31:57.815226    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:32:00.505732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.505703    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:32:00.632186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632148    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632206    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632240    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632436    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632482    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632549    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:00.632590 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632549    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632624    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632693    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.632878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632764    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:00.632967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632940    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:00.633035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.632964    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\") pod \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\" (UID: \"dc9da1c2-b651-4fec-bf7b-a11e6a407199\") "
Oct 30 12:32:00.633092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633054    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:00.633153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633127    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:00.633941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633913    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.633941 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633942    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.634140 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633967    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.634140 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633982    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.634140 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.633996    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.634764 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.634739    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:00.634993 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.634970    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:00.635203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.635185    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:32:00.635308 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.635290    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz" pod "dc9da1c2-b651-4fec-bf7b-a11e6a407199" (UID: "dc9da1c2-b651-4fec-bf7b-a11e6a407199"). InnerVolumeSpecName "kube-api-access-n7dvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:00.677655 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.677617    4453 generic.go:356] "Generic (PLEG): container finished" podID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerID="f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9" exitCode=0
Oct 30 12:32:00.677859 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.677761    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" event={"ID":"dc9da1c2-b651-4fec-bf7b-a11e6a407199","Type":"ContainerDied","Data":"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"}
Oct 30 12:32:00.677859 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.677843    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294" event={"ID":"dc9da1c2-b651-4fec-bf7b-a11e6a407199","Type":"ContainerDied","Data":"a20c39475abdd909a1545f2cc34d447f2fb2a1cd5a9e199a9f13b68755e4ee34"}
Oct 30 12:32:00.677985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.677883    4453 scope.go:122] "RemoveContainer" containerID="f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"
Oct 30 12:32:00.678073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.678059    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"
Oct 30 12:32:00.703466 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.703438    4453 scope.go:122] "RemoveContainer" containerID="f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"
Oct 30 12:32:00.703808 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:32:00.703785    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9\": container with ID starting with f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9 not found: ID does not exist" containerID="f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"
Oct 30 12:32:00.703868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.703818    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9"} err="failed to get container status \"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9\": rpc error: code = NotFound desc = could not find container \"f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9\": container with ID starting with f1065a31dc0685e92fe42fb7d2040fc45359d6345f408ebdd6f9cd3d6cd58cb9 not found: ID does not exist"
Oct 30 12:32:00.730382 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.730326    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:32:00.734137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.734064    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-28ca05-openshift-default-7567c7697c-79294"]
Oct 30 12:32:00.734977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.734957    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7dvz\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-kube-api-access-n7dvz\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.735031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.734980    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.735031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.734992    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.735031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.734999    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc9da1c2-b651-4fec-bf7b-a11e6a407199-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:00.916495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:00.916461    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" path="/var/lib/kubelet/pods/dc9da1c2-b651-4fec-bf7b-a11e6a407199/volumes"
Oct 30 12:32:06.515059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:06.515018    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:32:06.515705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:06.515250    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" podUID="bfdf710f-22a0-466a-9224-3b209c05f1d4" containerName="maas-api" containerID="cri-o://715ddbb8a42aa63854c6b7219af244af423e0ac80927576a8b6c1bf9ec55c5c7" gracePeriod=30
Oct 30 12:32:06.735894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:06.735854    4453 generic.go:356] "Generic (PLEG): container finished" podID="bfdf710f-22a0-466a-9224-3b209c05f1d4" containerID="715ddbb8a42aa63854c6b7219af244af423e0ac80927576a8b6c1bf9ec55c5c7" exitCode=0
Oct 30 12:32:06.736054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:06.735918    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" event={"ID":"bfdf710f-22a0-466a-9224-3b209c05f1d4","Type":"ContainerDied","Data":"715ddbb8a42aa63854c6b7219af244af423e0ac80927576a8b6c1bf9ec55c5c7"}
Oct 30 12:32:06.874087 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:06.874054    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:32:07.057802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.057764    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\") pod \"bfdf710f-22a0-466a-9224-3b209c05f1d4\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") "
Oct 30 12:32:07.058012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.057907    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") pod \"bfdf710f-22a0-466a-9224-3b209c05f1d4\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") "
Oct 30 12:32:07.058012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.058008    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\" (UniqueName: \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\") pod \"bfdf710f-22a0-466a-9224-3b209c05f1d4\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") "
Oct 30 12:32:07.058149 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.058052    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\") pod \"bfdf710f-22a0-466a-9224-3b209c05f1d4\" (UID: \"bfdf710f-22a0-466a-9224-3b209c05f1d4\") "
Oct 30 12:32:07.067558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.067522    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca" pod "bfdf710f-22a0-466a-9224-3b209c05f1d4" (UID: "bfdf710f-22a0-466a-9224-3b209c05f1d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:07.086154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.079085    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs" pod "bfdf710f-22a0-466a-9224-3b209c05f1d4" (UID: "bfdf710f-22a0-466a-9224-3b209c05f1d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:32:07.086154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.079228    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls" pod "bfdf710f-22a0-466a-9224-3b209c05f1d4" (UID: "bfdf710f-22a0-466a-9224-3b209c05f1d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:32:07.086154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.079422    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv" pod "bfdf710f-22a0-466a-9224-3b209c05f1d4" (UID: "bfdf710f-22a0-466a-9224-3b209c05f1d4"). InnerVolumeSpecName "kube-api-access-5g6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:07.161086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.161048    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g6kv\" (UniqueName: \"kubernetes.io/projected/bfdf710f-22a0-466a-9224-3b209c05f1d4-kube-api-access-5g6kv\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:07.161086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.161086    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf710f-22a0-466a-9224-3b209c05f1d4-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:07.161297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.161100    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:07.161297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.161113    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdf710f-22a0-466a-9224-3b209c05f1d4-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:07.572655 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:32:07.572616    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:32:07.744196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.744164    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"
Oct 30 12:32:07.744418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.744193    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s" event={"ID":"bfdf710f-22a0-466a-9224-3b209c05f1d4","Type":"ContainerDied","Data":"753b33968d617b9da92422809031cba290991b2b1f610f5c3914ced5f3332c6d"}
Oct 30 12:32:07.744418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.744255    4453 scope.go:122] "RemoveContainer" containerID="715ddbb8a42aa63854c6b7219af244af423e0ac80927576a8b6c1bf9ec55c5c7"
Oct 30 12:32:07.789098 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.789068    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:32:07.793159 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:07.793135    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d71fa-849f8c5466-wcc4s"]
Oct 30 12:32:08.913515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:08.913474    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdf710f-22a0-466a-9224-3b209c05f1d4" path="/var/lib/kubelet/pods/bfdf710f-22a0-466a-9224-3b209c05f1d4/volumes"
Oct 30 12:32:14.317180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:14.317083    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:32:14.317685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:14.317372    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy" containerID="cri-o://596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c" gracePeriod=30
Oct 30 12:32:14.326795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:14.326763    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:32:14.327008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:14.326983    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy" containerID="cri-o://c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124" gracePeriod=30
Oct 30 12:32:19.613237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.613212    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:32:19.652230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652204    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652251    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652292    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652325    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652443    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652568 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652550    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652684 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652635    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652733    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652834    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\") pod \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\" (UID: \"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a\") "
Oct 30 12:32:19.652912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.652887    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:19.653089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653070    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.653189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653084    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.653189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653157    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.653189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653014    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.653774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653741    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.653774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653771    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.653936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653787    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.653936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653801    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.653936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.653816    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.654785 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.654765    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.654977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.654952    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:32:19.655092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.655031    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "kube-api-access-hbm5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.655174 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.655145    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token" pod "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" (UID: "ff2e39cb-116d-416f-b45a-c55b3ed7ba8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.658633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.658613    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:32:19.755100 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755067    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755122    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755172    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755213    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755242    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755300 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755292    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755387    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755452    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755637 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755627    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\") pod \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\" (UID: \"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa\") "
Oct 30 12:32:19.755852 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.755797    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.756033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756011    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.756401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756376    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756406    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756421    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbm5w\" (UniqueName: \"kubernetes.io/projected/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-kube-api-access-hbm5w\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756439    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756453    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756466    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.756988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.756963    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.757291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.757266    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:32:19.757942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.757915    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.758135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.758113    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "kube-api-access-zplfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.758953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.758930    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:32:19.759046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.759011    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:32:19.760248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.760229    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy" pod "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" (UID: "1c33f649-f30b-49e6-8093-9a3bd5f9a1fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:32:19.853356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.853304    4453 generic.go:356] "Generic (PLEG): container finished" podID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerID="596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c" exitCode=0
Oct 30 12:32:19.853496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.853364    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"
Oct 30 12:32:19.853496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.853362    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" event={"ID":"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a","Type":"ContainerDied","Data":"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"}
Oct 30 12:32:19.853496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.853485    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm" event={"ID":"ff2e39cb-116d-416f-b45a-c55b3ed7ba8a","Type":"ContainerDied","Data":"c0a332911f2d0756ea27d97ef6270e8f992e1b0188398f12a461d50f2f816c93"}
Oct 30 12:32:19.853648 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.853516    4453 scope.go:122] "RemoveContainer" containerID="596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"
Oct 30 12:32:19.855049 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.855026    4453 generic.go:356] "Generic (PLEG): container finished" podID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerID="c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124" exitCode=0
Oct 30 12:32:19.855166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.855075    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"
Oct 30 12:32:19.855252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.855230    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" event={"ID":"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa","Type":"ContainerDied","Data":"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"}
Oct 30 12:32:19.855319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.855285    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh" event={"ID":"1c33f649-f30b-49e6-8093-9a3bd5f9a1fa","Type":"ContainerDied","Data":"40b735b9f07a23dbc08652cf6ef5ac3fc5a1539ef77bfbc098e52c456675720b"}
Oct 30 12:32:19.857461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857437    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplfn\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-kube-api-access-zplfn\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857460    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857470    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857478    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857486    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857494    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.857646 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.857502    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:32:19.865169 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.865140    4453 scope.go:122] "RemoveContainer" containerID="596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"
Oct 30 12:32:19.865509 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:32:19.865487    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c\": container with ID starting with 596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c not found: ID does not exist" containerID="596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"
Oct 30 12:32:19.865618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.865513    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c"} err="failed to get container status \"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c\": rpc error: code = NotFound desc = could not find container \"596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c\": container with ID starting with 596e12d43a810b2070229ee11c5002b757f6e00d1684ac4021f9fc157ef6557c not found: ID does not exist"
Oct 30 12:32:19.865618 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.865530    4453 scope.go:122] "RemoveContainer" containerID="c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"
Oct 30 12:32:19.875573 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.875553    4453 scope.go:122] "RemoveContainer" containerID="c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"
Oct 30 12:32:19.875833 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:32:19.875814    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124\": container with ID starting with c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124 not found: ID does not exist" containerID="c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"
Oct 30 12:32:19.875880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.875841    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124"} err="failed to get container status \"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124\": rpc error: code = NotFound desc = could not find container \"c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124\": container with ID starting with c583ecd6a3a76bdbd3e1237606ed114c76eda1658a7cd90c7fcc6175ef357124 not found: ID does not exist"
Oct 30 12:32:19.899306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.899283    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:32:19.903069 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.903050    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-dmzrm"]
Oct 30 12:32:19.930619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.930589    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:32:19.935455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:19.935432    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d71fa-openshift-default-7569b955d9-v4rbh"]
Oct 30 12:32:20.915290 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:20.915251    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" path="/var/lib/kubelet/pods/1c33f649-f30b-49e6-8093-9a3bd5f9a1fa/volumes"
Oct 30 12:32:20.916233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:20.916209    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" path="/var/lib/kubelet/pods/ff2e39cb-116d-416f-b45a-c55b3ed7ba8a/volumes"
Oct 30 12:32:37.572323 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:32:37.572286    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:32:48.738316 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.738280    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:32:48.739095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739078    4453 cpu_manager.go:414] "containerMap: removing container" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739096    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739108    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739145 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739114    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739127    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739172    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739186    4453 cpu_manager.go:414] "containerMap: removing container" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739190    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739197    4453 cpu_manager.go:414] "containerMap: removing container" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739202    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739212    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739216    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739225    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739235    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739244    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739249    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739257    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739261    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739269    4453 cpu_manager.go:414] "containerMap: removing container" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739274    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739284    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739281 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739289    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739298    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739302    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739307    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739311    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739317    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739322    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739342    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739347    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739358    4453 cpu_manager.go:414] "containerMap: removing container" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739363    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739377    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739382    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739391    4453 cpu_manager.go:414] "containerMap: removing container" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739396    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739402    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739407    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739413    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739417    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739422    4453 cpu_manager.go:414] "containerMap: removing container" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739426    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739435    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739440    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739448    4453 cpu_manager.go:414] "containerMap: removing container" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739452    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739458    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739462    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739471    4453 cpu_manager.go:414] "containerMap: removing container" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739476    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739485    4453 cpu_manager.go:414] "containerMap: removing container" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739512    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739523    4453 cpu_manager.go:414] "containerMap: removing container" podUID="bfdf710f-22a0-466a-9224-3b209c05f1d4" containerName="maas-api"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739529    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdf710f-22a0-466a-9224-3b209c05f1d4" containerName="maas-api"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739539    4453 cpu_manager.go:414] "containerMap: removing container" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739543    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739550    4453 cpu_manager.go:414] "containerMap: removing container" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739555    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739565    4453 cpu_manager.go:414] "containerMap: removing container" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739569    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739580    4453 cpu_manager.go:414] "containerMap: removing container" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739585    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739718    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739726    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739732    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739739    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2e39cb-116d-416f-b45a-c55b3ed7ba8a" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739746    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739751    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739757    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739764    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffea7c47-cd3e-4738-928c-3a680d656825" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739771    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739777    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739784    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739789    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c33f649-f30b-49e6-8093-9a3bd5f9a1fa" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739796    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739803    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739809    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739813    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739819    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739826    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7cecb5-1996-4ce0-92f2-21ac8c2339d8" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739833    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="87cda50b-fdb2-4b11-913e-544644360e39" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739838    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739843    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739849    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739854    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e50b525-461a-40a4-aa8e-e5bafaa95f6e" containerName="payload-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739861    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739866    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739871    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c911504-d363-4f1f-a526-3f98f4810061" containerName="payload-pre-processing"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739877    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aed1b94-6670-47f0-92eb-5d340291567f" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739883    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739890    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdf710f-22a0-466a-9224-3b209c05f1d4" containerName="maas-api"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739897    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739902    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739908    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9da1c2-b651-4fec-bf7b-a11e6a407199" containerName="istio-proxy"
Oct 30 12:32:48.739939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.739914    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="20717866-3b47-441f-8deb-e1f810cb9c43" containerName="payload-pre-processing"
Oct 30 12:32:48.743704 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.743688    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761082    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761190    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761357 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761314    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761389    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761489 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761446    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761573 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761555    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjjx\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761602    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761724 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761696    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.761786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.761748    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.768663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.768634    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:32:48.862528 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.862492    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjjx\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.862719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.862543    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.862813 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.862794    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.862880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.862846    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.862999    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863057    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863164    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863192    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863224    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863414    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863460    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863557 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863533    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863597 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863541    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.863636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.863597    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.865431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.865404    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.866023 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.866002    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.887560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.887536    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:48.887723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:48.887705    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjjx\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tjjd2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:49.064287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:49.064215    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:49.389192 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:49.387772    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:32:49.392642 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:32:49.392611    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8570a056_36cc_4e15_81b1_8d5527f3a7a2.slice/crio-e8b225a59dd49566e79bbc95d11783c497bd5721912d28e0cd768498cf1f1c2a WatchSource:0}: Error finding container e8b225a59dd49566e79bbc95d11783c497bd5721912d28e0cd768498cf1f1c2a: Status 404 returned error can't find the container with id e8b225a59dd49566e79bbc95d11783c497bd5721912d28e0cd768498cf1f1c2a
Oct 30 12:32:49.395399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:49.395364    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:32:49.395484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:49.395463    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:32:49.395533 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:49.395491    4453 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 12:32:50.228346 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:50.228293    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" event={"ID":"8570a056-36cc-4e15-81b1-8d5527f3a7a2","Type":"ContainerStarted","Data":"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"}
Oct 30 12:32:50.228811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:50.228383    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" event={"ID":"8570a056-36cc-4e15-81b1-8d5527f3a7a2","Type":"ContainerStarted","Data":"e8b225a59dd49566e79bbc95d11783c497bd5721912d28e0cd768498cf1f1c2a"}
Oct 30 12:32:50.270370 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:50.270116    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podStartSLOduration=2.270097863 podStartE2EDuration="2.270097863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:32:50.268127461 +0000 UTC m=+3769.488412371" watchObservedRunningTime="2026-10-30 12:32:50.270097863 +0000 UTC m=+3769.490382776"
Oct 30 12:32:51.065307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:51.065276    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:51.067390 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:51.067359    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:51.067520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:51.067411    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:52.065137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:52.065103    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:52.065535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:52.065163    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:53.065133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:53.065098    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:53.065320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:53.065155    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:54.066728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:54.066691    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:54.067356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:54.067300    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:55.065017 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:55.064980    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:55.065202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:55.065035    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:56.065345 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:56.065298    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:56.065750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:56.065381    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:57.064592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:57.064552    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 12:32:57.064761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:57.064619    4453 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 12:32:58.264179 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:58.264143    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:58.264708 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:58.264674    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:32:58.265605 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:32:58.265584    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:33:07.571637 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:33:07.571594    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:33:37.571547 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:33:37.571511    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:33:56.648282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:33:56.648200    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:34:01.649290 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:01.649242    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing" containerID="cri-o://81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10" gracePeriod=25
Oct 30 12:34:03.988432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:03.988390    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:07.571990 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:34:07.571952    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:34:13.234098 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.234060    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:34:13.235238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.235203    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerName="maas-api" containerID="cri-o://1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c" gracePeriod=30
Oct 30 12:34:13.587582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.587553    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:34:13.669053 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.669011    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\") pod \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") "
Oct 30 12:34:13.686110 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.686078    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\") pod \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") "
Oct 30 12:34:13.686776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.686753    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\" (UniqueName: \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\") pod \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") "
Oct 30 12:34:13.687713 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.687673    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca" pod "0a23799d-4c42-4f57-be2a-21c1067e6e2e" (UID: "0a23799d-4c42-4f57-be2a-21c1067e6e2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:13.687849 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.687833    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\") pod \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\" (UID: \"0a23799d-4c42-4f57-be2a-21c1067e6e2e\") "
Oct 30 12:34:13.688748 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.688724    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a23799d-4c42-4f57-be2a-21c1067e6e2e-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:13.757156 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.757086    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls" pod "0a23799d-4c42-4f57-be2a-21c1067e6e2e" (UID: "0a23799d-4c42-4f57-be2a-21c1067e6e2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:13.767949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.767882    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh" pod "0a23799d-4c42-4f57-be2a-21c1067e6e2e" (UID: "0a23799d-4c42-4f57-be2a-21c1067e6e2e"). InnerVolumeSpecName "kube-api-access-mvtkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:13.767949 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.767930    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs" pod "0a23799d-4c42-4f57-be2a-21c1067e6e2e" (UID: "0a23799d-4c42-4f57-be2a-21c1067e6e2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:34:13.858400 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.858365    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:13.858400 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.858396    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a23799d-4c42-4f57-be2a-21c1067e6e2e-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:13.858603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.858410    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtkh\" (UniqueName: \"kubernetes.io/projected/0a23799d-4c42-4f57-be2a-21c1067e6e2e-kube-api-access-mvtkh\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:13.968992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.968948    4453 generic.go:356] "Generic (PLEG): container finished" podID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerID="1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c" exitCode=0
Oct 30 12:34:13.968992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.968991    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"
Oct 30 12:34:13.969207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.969024    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" event={"ID":"0a23799d-4c42-4f57-be2a-21c1067e6e2e","Type":"ContainerDied","Data":"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"}
Oct 30 12:34:13.969207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.969073    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" event={"ID":"0a23799d-4c42-4f57-be2a-21c1067e6e2e","Type":"ContainerDied","Data":"5b48a6d00c25943abcec792ef028d244dcf3fc4bb3032e4144af420be8b7b407"}
Oct 30 12:34:13.969207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.969095    4453 scope.go:122] "RemoveContainer" containerID="1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"
Oct 30 12:34:13.982647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.982606    4453 scope.go:122] "RemoveContainer" containerID="1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"
Oct 30 12:34:13.982923 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:34:13.982903    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c\": container with ID starting with 1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c not found: ID does not exist" containerID="1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"
Oct 30 12:34:13.982980 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.982935    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c"} err="failed to get container status \"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c\": rpc error: code = NotFound desc = could not find container \"1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c\": container with ID starting with 1c7d7ece462dbbd5c29158128b04f811a7db4fecf1fbecc64a0760ab33072c2c not found: ID does not exist"
Oct 30 12:34:13.988168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:13.988143    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:14.021576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:14.021554    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:34:14.025156 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:14.025131    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx"]
Oct 30 12:34:14.913579 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:14.913546    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" path="/var/lib/kubelet/pods/0a23799d-4c42-4f57-be2a-21c1067e6e2e/volumes"
Oct 30 12:34:16.427741 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:16.427698    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae3538-5c885b876d-fznvx" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.163:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:34:21.186257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:21.186211    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:34:21.186800 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:21.186464    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy" containerID="cri-o://81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26" gracePeriod=30
Oct 30 12:34:21.938831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:21.938803    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:34:22.032284 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.032248    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\") pod \"1acacdad-31aa-4fdc-aea1-83197367f134\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") "
Oct 30 12:34:22.032284 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.032285    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\" (UniqueName: \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\") pod \"1acacdad-31aa-4fdc-aea1-83197367f134\" (UID: \"1acacdad-31aa-4fdc-aea1-83197367f134\") "
Oct 30 12:34:22.032712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.032687    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume" pod "1acacdad-31aa-4fdc-aea1-83197367f134" (UID: "1acacdad-31aa-4fdc-aea1-83197367f134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:22.033072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.033052    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acacdad-31aa-4fdc-aea1-83197367f134-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:22.034793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.034768    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2" pod "1acacdad-31aa-4fdc-aea1-83197367f134" (UID: "1acacdad-31aa-4fdc-aea1-83197367f134"). InnerVolumeSpecName "kube-api-access-zd2t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:22.057212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.057180    4453 generic.go:356] "Generic (PLEG): container finished" podID="1acacdad-31aa-4fdc-aea1-83197367f134" containerID="81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10" exitCode=0
Oct 30 12:34:22.057321 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.057243    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"
Oct 30 12:34:22.057321 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.057270    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" event={"ID":"1acacdad-31aa-4fdc-aea1-83197367f134","Type":"ContainerDied","Data":"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"}
Oct 30 12:34:22.057458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.057364    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc" event={"ID":"1acacdad-31aa-4fdc-aea1-83197367f134","Type":"ContainerDied","Data":"121b24a82ffadd0282d460c46e04cd81bd9869e404a1e631577653e821bc5736"}
Oct 30 12:34:22.057458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.057369    4453 scope.go:122] "RemoveContainer" containerID="81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"
Oct 30 12:34:22.068279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.068262    4453 scope.go:122] "RemoveContainer" containerID="81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"
Oct 30 12:34:22.068579 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:34:22.068561    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10\": container with ID starting with 81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10 not found: ID does not exist" containerID="81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"
Oct 30 12:34:22.068633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.068586    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10"} err="failed to get container status \"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10\": rpc error: code = NotFound desc = could not find container \"81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10\": container with ID starting with 81c4f52eb16577ecf27d1433dec812afca89f277a4a30ecba098c8aee6a22c10 not found: ID does not exist"
Oct 30 12:34:22.104305 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.104282    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:34:22.109639 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.109613    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ae3538-6c6b77b5fd-ks9xc"]
Oct 30 12:34:22.134538 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.134512    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2t2\" (UniqueName: \"kubernetes.io/projected/1acacdad-31aa-4fdc-aea1-83197367f134-kube-api-access-zd2t2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:22.914347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:22.914310    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" path="/var/lib/kubelet/pods/1acacdad-31aa-4fdc-aea1-83197367f134/volumes"
Oct 30 12:34:26.493041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.493012    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:34:26.605863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.605819    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.605921    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.605969    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606053    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606094    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606126    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606220    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606269    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606287    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606319    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\") pod \"f2931ba8-16df-4903-bfa9-8cf801e80efe\" (UID: \"f2931ba8-16df-4903-bfa9-8cf801e80efe\") "
Oct 30 12:34:26.606626 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606441    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.606797 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606744    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.606896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.606871    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.607052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.607031    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:26.607122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.607104    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.607183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.607130    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.607183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.607145    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.607183 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.607159    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.608614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.608536    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:34:26.608614 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.608590    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:26.608743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.608692    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "kube-api-access-htr9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:26.609088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.609070    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy" pod "f2931ba8-16df-4903-bfa9-8cf801e80efe" (UID: "f2931ba8-16df-4903-bfa9-8cf801e80efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:34:26.708691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.708658    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htr9b\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-kube-api-access-htr9b\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.708691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.708688    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.708691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.708697    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.708933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.708705    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2931ba8-16df-4903-bfa9-8cf801e80efe-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:26.708933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:26.708713    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2931ba8-16df-4903-bfa9-8cf801e80efe-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:27.114012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.113969    4453 generic.go:356] "Generic (PLEG): container finished" podID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerID="81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26" exitCode=0
Oct 30 12:34:27.114188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.114040    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"
Oct 30 12:34:27.114188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.114045    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" event={"ID":"f2931ba8-16df-4903-bfa9-8cf801e80efe","Type":"ContainerDied","Data":"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"}
Oct 30 12:34:27.114188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.114134    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck" event={"ID":"f2931ba8-16df-4903-bfa9-8cf801e80efe","Type":"ContainerDied","Data":"3db3d70c4c4f52f4e4ee3f505d7465d58330ad8b97d754390442131f4691e4d5"}
Oct 30 12:34:27.114188 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.114165    4453 scope.go:122] "RemoveContainer" containerID="81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"
Oct 30 12:34:27.127418 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.127394    4453 scope.go:122] "RemoveContainer" containerID="81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"
Oct 30 12:34:27.127735 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:34:27.127718    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26\": container with ID starting with 81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26 not found: ID does not exist" containerID="81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"
Oct 30 12:34:27.127809 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.127745    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26"} err="failed to get container status \"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26\": rpc error: code = NotFound desc = could not find container \"81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26\": container with ID starting with 81bab343936c80c13b5a37b3d31be06394585fda48a7bb1e971102fa91036b26 not found: ID does not exist"
Oct 30 12:34:27.160730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.160676    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:34:27.168238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:27.168203    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ae3538-openshift-default-84f8bb6f-2vmck"]
Oct 30 12:34:28.913062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:28.913027    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" path="/var/lib/kubelet/pods/f2931ba8-16df-4903-bfa9-8cf801e80efe/volumes"
Oct 30 12:34:32.295933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:32.295896    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:34:37.302016 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:37.301969    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing" containerID="cri-o://4f503b519cdc5ac8e82c238664320b22bed7587e2e467f446dc750aa8c08f6d8" gracePeriod=25
Oct 30 12:34:37.572284 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:34:37.572197    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:34:46.054115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:46.054073    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:56.054414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:56.054376    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:34:57.342393 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.342359    4453 generic.go:356] "Generic (PLEG): container finished" podID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerID="4f503b519cdc5ac8e82c238664320b22bed7587e2e467f446dc750aa8c08f6d8" exitCode=0
Oct 30 12:34:57.342753 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.342438    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" event={"ID":"3b4fb54c-b6ac-48c3-addd-4378b0524b77","Type":"ContainerDied","Data":"4f503b519cdc5ac8e82c238664320b22bed7587e2e467f446dc750aa8c08f6d8"}
Oct 30 12:34:57.490218 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.490186    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:34:57.553122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.553085    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\" (UniqueName: \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\") pod \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") "
Oct 30 12:34:57.553359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.553189    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\") pod \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\" (UID: \"3b4fb54c-b6ac-48c3-addd-4378b0524b77\") "
Oct 30 12:34:57.553939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.553690    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume" pod "3b4fb54c-b6ac-48c3-addd-4378b0524b77" (UID: "3b4fb54c-b6ac-48c3-addd-4378b0524b77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:34:57.553939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.553906    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4fb54c-b6ac-48c3-addd-4378b0524b77-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:57.555554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.555527    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5" pod "3b4fb54c-b6ac-48c3-addd-4378b0524b77" (UID: "3b4fb54c-b6ac-48c3-addd-4378b0524b77"). InnerVolumeSpecName "kube-api-access-xk8w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:34:57.655131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:57.655102    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8w5\" (UniqueName: \"kubernetes.io/projected/3b4fb54c-b6ac-48c3-addd-4378b0524b77-kube-api-access-xk8w5\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:34:58.353894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.353858    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm" event={"ID":"3b4fb54c-b6ac-48c3-addd-4378b0524b77","Type":"ContainerDied","Data":"dfd6e5a81f444d0f3cdd88bd023e9bf46377c75882874b45bde1a33ef896f3b6"}
Oct 30 12:34:58.353894 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.353895    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"
Oct 30 12:34:58.354545 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.353933    4453 scope.go:122] "RemoveContainer" containerID="4f503b519cdc5ac8e82c238664320b22bed7587e2e467f446dc750aa8c08f6d8"
Oct 30 12:34:58.428494 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.428457    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:34:58.432018 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.431988    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0ecae0-69fb4b6c94-4fjnm"]
Oct 30 12:34:58.914296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:34:58.914257    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" path="/var/lib/kubelet/pods/3b4fb54c-b6ac-48c3-addd-4378b0524b77/volumes"
Oct 30 12:35:07.572175 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:35:07.572133    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:35:18.880070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:18.879854    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:35:18.880834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:18.880105    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy" containerID="cri-o://618fbc7de1663990e7b5bd1f63fb10e274a4138369c79f430f9f1c60986f7487" gracePeriod=30
Oct 30 12:35:24.554401 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.554370    4453 generic.go:356] "Generic (PLEG): container finished" podID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerID="618fbc7de1663990e7b5bd1f63fb10e274a4138369c79f430f9f1c60986f7487" exitCode=0
Oct 30 12:35:24.554751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.554443    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerDied","Data":"618fbc7de1663990e7b5bd1f63fb10e274a4138369c79f430f9f1c60986f7487"}
Oct 30 12:35:24.554751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.554503    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5" event={"ID":"be9525f0-ff85-42d9-a124-e7c76c4d121f","Type":"ContainerDied","Data":"ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be"}
Oct 30 12:35:24.554751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.554520    4453 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddff74f7eda246764c6e4da251a6b9402dae40b914ae1e623da471d36ea025be"
Oct 30 12:35:24.554751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.554534    4453 scope.go:122] "RemoveContainer" containerID="08d2cd3d089fd401e7dd0b88d204f51aa65945d80704d8d893c04ef288e58846"
Oct 30 12:35:24.576986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.576967    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:35:24.684146 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684064    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684170    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684232    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684300    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684348    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684434    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684471    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684498    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.684650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684594    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:24.684883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684649    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:24.684883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684678    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:24.684883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684707    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:24.685049 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.684894    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:24.685049 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685028    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\") pod \"be9525f0-ff85-42d9-a124-e7c76c4d121f\" (UID: \"be9525f0-ff85-42d9-a124-e7c76c4d121f\") "
Oct 30 12:35:24.685703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685675    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.685703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685704    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.686173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685716    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be9525f0-ff85-42d9-a124-e7c76c4d121f-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.686173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685725    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.686173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.685734    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.686718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.686686    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:35:24.686944 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.686923    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "kube-api-access-dhfjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:24.687045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.687028    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:35:24.687262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.687241    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token" pod "be9525f0-ff85-42d9-a124-e7c76c4d121f" (UID: "be9525f0-ff85-42d9-a124-e7c76c4d121f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:24.787388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.787366    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.787388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.787387    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfjv\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-kube-api-access-dhfjv\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.787563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.787397    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:24.787563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:24.787406    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be9525f0-ff85-42d9-a124-e7c76c4d121f-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:25.563535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:25.563496    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"
Oct 30 12:35:25.647407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:25.647373    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:35:25.651657 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:25.651630    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0ecae0-openshift-default-dcc48f8b5-2bnk5"]
Oct 30 12:35:26.913629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:26.913599    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" path="/var/lib/kubelet/pods/be9525f0-ff85-42d9-a124-e7c76c4d121f/volumes"
Oct 30 12:35:28.410727 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:28.410693    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:35:28.444442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:28.444405    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:35:33.411389 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.411320    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" podUID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" containerName="payload-processing" containerID="cri-o://c5cd0dde83d28018cd19e5f21de1c793fcd9255ff904e710122591394881ae93" gracePeriod=25
Oct 30 12:35:33.445298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.445259    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing" containerID="cri-o://52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef" gracePeriod=25
Oct 30 12:35:33.625443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.625227    4453 generic.go:356] "Generic (PLEG): container finished" podID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" containerID="c5cd0dde83d28018cd19e5f21de1c793fcd9255ff904e710122591394881ae93" exitCode=0
Oct 30 12:35:33.625443 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.625358    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" event={"ID":"828dc1ab-5579-4734-b6ae-3cd75432c9dd","Type":"ContainerDied","Data":"c5cd0dde83d28018cd19e5f21de1c793fcd9255ff904e710122591394881ae93"}
Oct 30 12:35:33.695118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.695091    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:35:33.734948 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.734914    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\") pod \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") "
Oct 30 12:35:33.735135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.735072    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\" (UniqueName: \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\") pod \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\" (UID: \"828dc1ab-5579-4734-b6ae-3cd75432c9dd\") "
Oct 30 12:35:33.735474 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.735450    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume" pod "828dc1ab-5579-4734-b6ae-3cd75432c9dd" (UID: "828dc1ab-5579-4734-b6ae-3cd75432c9dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:33.735715 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.735670    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/828dc1ab-5579-4734-b6ae-3cd75432c9dd-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:33.737761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.737735    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49" pod "828dc1ab-5579-4734-b6ae-3cd75432c9dd" (UID: "828dc1ab-5579-4734-b6ae-3cd75432c9dd"). InnerVolumeSpecName "kube-api-access-4rh49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:33.836962 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:33.836927    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rh49\" (UniqueName: \"kubernetes.io/projected/828dc1ab-5579-4734-b6ae-3cd75432c9dd-kube-api-access-4rh49\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:34.633610 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.633577    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"
Oct 30 12:35:34.634134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.633583    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q" event={"ID":"828dc1ab-5579-4734-b6ae-3cd75432c9dd","Type":"ContainerDied","Data":"4fef4215b6894f5cb0c68a94affb52befd0e980db1044dac113e10d88adbc148"}
Oct 30 12:35:34.634134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.633740    4453 scope.go:122] "RemoveContainer" containerID="c5cd0dde83d28018cd19e5f21de1c793fcd9255ff904e710122591394881ae93"
Oct 30 12:35:34.678057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.678027    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:35:34.686743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.686714    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0190d8-7fc6795d84-mc66q"]
Oct 30 12:35:34.913587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:34.913495    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" path="/var/lib/kubelet/pods/828dc1ab-5579-4734-b6ae-3cd75432c9dd/volumes"
Oct 30 12:35:37.571763 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:35:37.571726    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:35:37.921389 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:37.921354    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:35:42.954675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:42.954635    4453 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 12:35:42.955315 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:42.955268    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-hpp9t" podUID="c5ef9f10-6825-4798-9579-2d1f3fed2dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 12:35:47.921706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:47.921663    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:35:53.729449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.729423    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:35:53.844514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.844479    4453 generic.go:356] "Generic (PLEG): container finished" podID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerID="52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef" exitCode=0
Oct 30 12:35:53.844725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.844527    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"
Oct 30 12:35:53.844725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.844566    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" event={"ID":"f8443e82-cb7c-4042-a03f-0e51d8160358","Type":"ContainerDied","Data":"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"}
Oct 30 12:35:53.844725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.844617    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f" event={"ID":"f8443e82-cb7c-4042-a03f-0e51d8160358","Type":"ContainerDied","Data":"6cd44dcb050c4099f6973a7718edc17fd7332f7ca01c07a4925cbb82f1fe537c"}
Oct 30 12:35:53.844725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.844640    4453 scope.go:122] "RemoveContainer" containerID="52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"
Oct 30 12:35:53.853078 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.853050    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\" (UniqueName: \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\") pod \"f8443e82-cb7c-4042-a03f-0e51d8160358\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") "
Oct 30 12:35:53.853211 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.853130    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\") pod \"f8443e82-cb7c-4042-a03f-0e51d8160358\" (UID: \"f8443e82-cb7c-4042-a03f-0e51d8160358\") "
Oct 30 12:35:53.853539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.853512    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume" pod "f8443e82-cb7c-4042-a03f-0e51d8160358" (UID: "f8443e82-cb7c-4042-a03f-0e51d8160358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:35:53.854060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.854037    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8443e82-cb7c-4042-a03f-0e51d8160358-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:53.855051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.855031    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c" pod "f8443e82-cb7c-4042-a03f-0e51d8160358" (UID: "f8443e82-cb7c-4042-a03f-0e51d8160358"). InnerVolumeSpecName "kube-api-access-jjb6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:35:53.855297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.855279    4453 scope.go:122] "RemoveContainer" containerID="52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"
Oct 30 12:35:53.855598 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:35:53.855578    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef\": container with ID starting with 52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef not found: ID does not exist" containerID="52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"
Oct 30 12:35:53.855670 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.855605    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef"} err="failed to get container status \"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef\": rpc error: code = NotFound desc = could not find container \"52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef\": container with ID starting with 52c0cecef8d18cc847c45796c5c4f17b3021182d871ff327155db4e150f9d0ef not found: ID does not exist"
Oct 30 12:35:53.955511 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:53.955484    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjb6c\" (UniqueName: \"kubernetes.io/projected/f8443e82-cb7c-4042-a03f-0e51d8160358-kube-api-access-jjb6c\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:35:54.189204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:54.189161    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:35:54.194863 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:54.194834    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0190d8-7777999979-z2k5f"]
Oct 30 12:35:54.912910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:35:54.912876    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" path="/var/lib/kubelet/pods/f8443e82-cb7c-4042-a03f-0e51d8160358/volumes"
Oct 30 12:36:02.369649 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:02.369608    4453 scope.go:122] "RemoveContainer" containerID="618fbc7de1663990e7b5bd1f63fb10e274a4138369c79f430f9f1c60986f7487"
Oct 30 12:36:03.334633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:03.334592    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:36:03.368068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:03.368033    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:36:07.571720 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:07.571684    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:36:08.335887 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:08.335830    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing" containerID="cri-o://9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b" gracePeriod=25
Oct 30 12:36:08.368323 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:08.368285    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing" containerID="cri-o://55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4" gracePeriod=25
Oct 30 12:36:16.339456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:16.339411    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:16.340386 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:16.340356    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:26.316797 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:26.316752    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:26.317313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:26.317025    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:36:28.655737 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.655712    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:36:28.676158 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.676131    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:36:28.684856 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.684835    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\") pod \"872e9968-a3f9-4637-ac74-62eee04586d3\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") "
Oct 30 12:36:28.684963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.684931    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\" (UniqueName: \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\") pod \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") "
Oct 30 12:36:28.685008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.684959    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\") pod \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\" (UID: \"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b\") "
Oct 30 12:36:28.685166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.685147    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\" (UniqueName: \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\") pod \"872e9968-a3f9-4637-ac74-62eee04586d3\" (UID: \"872e9968-a3f9-4637-ac74-62eee04586d3\") "
Oct 30 12:36:28.685240 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.685178    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume" pod "872e9968-a3f9-4637-ac74-62eee04586d3" (UID: "872e9968-a3f9-4637-ac74-62eee04586d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:28.685432 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.685410    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume" pod "dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" (UID: "dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:28.685699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.685670    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872e9968-a3f9-4637-ac74-62eee04586d3-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:28.685699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.685695    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:28.687139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.687119    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq" pod "dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" (UID: "dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b"). InnerVolumeSpecName "kube-api-access-tk4cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:28.687196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.687153    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx" pod "872e9968-a3f9-4637-ac74-62eee04586d3" (UID: "872e9968-a3f9-4637-ac74-62eee04586d3"). InnerVolumeSpecName "kube-api-access-6vzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:28.786459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.786429    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk4cq\" (UniqueName: \"kubernetes.io/projected/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b-kube-api-access-tk4cq\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:28.786459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:28.786455    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vzzx\" (UniqueName: \"kubernetes.io/projected/872e9968-a3f9-4637-ac74-62eee04586d3-kube-api-access-6vzzx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:29.204700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.204617    4453 generic.go:356] "Generic (PLEG): container finished" podID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerID="55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4" exitCode=0
Oct 30 12:36:29.204700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.204660    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"
Oct 30 12:36:29.204902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.204695    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" event={"ID":"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b","Type":"ContainerDied","Data":"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"}
Oct 30 12:36:29.204902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.204747    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4" event={"ID":"dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b","Type":"ContainerDied","Data":"ce886dfb1f54bfea4b602bb8c6d0614bd7fea6f26064c5941dacd1a8b9dfda7f"}
Oct 30 12:36:29.204902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.204778    4453 scope.go:122] "RemoveContainer" containerID="55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"
Oct 30 12:36:29.217622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.217590    4453 generic.go:356] "Generic (PLEG): container finished" podID="872e9968-a3f9-4637-ac74-62eee04586d3" containerID="9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b" exitCode=0
Oct 30 12:36:29.217795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.217773    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"
Oct 30 12:36:29.218003 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.217776    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" event={"ID":"872e9968-a3f9-4637-ac74-62eee04586d3","Type":"ContainerDied","Data":"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"}
Oct 30 12:36:29.218126 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.218069    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk" event={"ID":"872e9968-a3f9-4637-ac74-62eee04586d3","Type":"ContainerDied","Data":"1e7b335cc92a0b3afe38cb1d8074c5021c9aae246a31953da858528714cc7564"}
Oct 30 12:36:29.228314 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.228295    4453 scope.go:122] "RemoveContainer" containerID="55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"
Oct 30 12:36:29.228591 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:29.228570    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4\": container with ID starting with 55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4 not found: ID does not exist" containerID="55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"
Oct 30 12:36:29.228659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.228598    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4"} err="failed to get container status \"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4\": rpc error: code = NotFound desc = could not find container \"55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4\": container with ID starting with 55fa8dfe651c0525d64593718bf743d593dcb9224e3a6afc9d26e84d041f93c4 not found: ID does not exist"
Oct 30 12:36:29.228659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.228614    4453 scope.go:122] "RemoveContainer" containerID="9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"
Oct 30 12:36:29.237767 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.237738    4453 scope.go:122] "RemoveContainer" containerID="9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"
Oct 30 12:36:29.238069 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:29.238049    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b\": container with ID starting with 9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b not found: ID does not exist" containerID="9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"
Oct 30 12:36:29.238138 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.238076    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b"} err="failed to get container status \"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b\": rpc error: code = NotFound desc = could not find container \"9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b\": container with ID starting with 9299cf26cd25d2d73ac8e952e2ef2bb6d480a829324b81db3e344e819805308b not found: ID does not exist"
Oct 30 12:36:29.269805 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.269780    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:36:29.275169 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.275147    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-31a6e3-65bcc7f487-kkcp4"]
Oct 30 12:36:29.307356 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.307315    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:36:29.315122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:29.315097    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-31a6e3-5b44896d5d-mgpjk"]
Oct 30 12:36:30.913026 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:30.912993    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" path="/var/lib/kubelet/pods/872e9968-a3f9-4637-ac74-62eee04586d3/volumes"
Oct 30 12:36:30.913433 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:30.913417    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" path="/var/lib/kubelet/pods/dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b/volumes"
Oct 30 12:36:37.570970 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:37.570923    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:36:39.895086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:39.895047    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:36:39.895711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:39.895308    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" podUID="4362bb18-c55b-4326-8208-90b80ab4571f" containerName="maas-api" containerID="cri-o://6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e" gracePeriod=30
Oct 30 12:36:40.228598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.228569    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:36:40.401697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.401658    4453 generic.go:356] "Generic (PLEG): container finished" podID="4362bb18-c55b-4326-8208-90b80ab4571f" containerID="6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e" exitCode=0
Oct 30 12:36:40.401903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.401824    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" event={"ID":"4362bb18-c55b-4326-8208-90b80ab4571f","Type":"ContainerDied","Data":"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"}
Oct 30 12:36:40.401903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.401874    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf" event={"ID":"4362bb18-c55b-4326-8208-90b80ab4571f","Type":"ContainerDied","Data":"8b1f69ba262faa703f14daf3eb1966aaa8b2c1675bdc0761cad350badb71f0f8"}
Oct 30 12:36:40.402009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.401912    4453 scope.go:122] "RemoveContainer" containerID="6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"
Oct 30 12:36:40.402088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.402076    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"
Oct 30 12:36:40.415446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.415414    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\") pod \"4362bb18-c55b-4326-8208-90b80ab4571f\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") "
Oct 30 12:36:40.415643 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.415506    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\" (UniqueName: \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\") pod \"4362bb18-c55b-4326-8208-90b80ab4571f\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") "
Oct 30 12:36:40.416248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.416219    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\") pod \"4362bb18-c55b-4326-8208-90b80ab4571f\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") "
Oct 30 12:36:40.416400 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.416286    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\") pod \"4362bb18-c55b-4326-8208-90b80ab4571f\" (UID: \"4362bb18-c55b-4326-8208-90b80ab4571f\") "
Oct 30 12:36:40.417135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.417102    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca" pod "4362bb18-c55b-4326-8208-90b80ab4571f" (UID: "4362bb18-c55b-4326-8208-90b80ab4571f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:40.418700 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.418677    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs" pod "4362bb18-c55b-4326-8208-90b80ab4571f" (UID: "4362bb18-c55b-4326-8208-90b80ab4571f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:36:40.418868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.418848    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746" pod "4362bb18-c55b-4326-8208-90b80ab4571f" (UID: "4362bb18-c55b-4326-8208-90b80ab4571f"). InnerVolumeSpecName "kube-api-access-wk746". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:40.421270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.420737    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls" pod "4362bb18-c55b-4326-8208-90b80ab4571f" (UID: "4362bb18-c55b-4326-8208-90b80ab4571f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:36:40.424017 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.423993    4453 scope.go:122] "RemoveContainer" containerID="6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"
Oct 30 12:36:40.424434 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:40.424408    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e\": container with ID starting with 6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e not found: ID does not exist" containerID="6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"
Oct 30 12:36:40.424531 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.424444    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e"} err="failed to get container status \"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e\": rpc error: code = NotFound desc = could not find container \"6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e\": container with ID starting with 6af565f4e34de651f000a1e3af25680604cd05ec05ef90a130a02cceae4fd24e not found: ID does not exist"
Oct 30 12:36:40.517854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.517778    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:40.517854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.517804    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk746\" (UniqueName: \"kubernetes.io/projected/4362bb18-c55b-4326-8208-90b80ab4571f-kube-api-access-wk746\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:40.517854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.517814    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4362bb18-c55b-4326-8208-90b80ab4571f-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:40.517854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.517823    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4362bb18-c55b-4326-8208-90b80ab4571f-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:40.748712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.748656    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:36:40.752173 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.752136    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31a6e3-74b5db669d-hs6rf"]
Oct 30 12:36:40.913261 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:40.913222    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4362bb18-c55b-4326-8208-90b80ab4571f" path="/var/lib/kubelet/pods/4362bb18-c55b-4326-8208-90b80ab4571f/volumes"
Oct 30 12:36:48.444011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:48.443807    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:36:48.444485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:48.444080    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy" containerID="cri-o://a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573" gracePeriod=30
Oct 30 12:36:53.738104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.738074    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:36:53.795789 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.795745    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.795799    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.795827    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796008 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.795983    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.796039    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.796074    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.796201    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796272 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.796242    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.796449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.796299    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\") pod \"5809f1df-91c9-484a-b1b2-25c4434c5a82\" (UID: \"5809f1df-91c9-484a-b1b2-25c4434c5a82\") "
Oct 30 12:36:53.797421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.797390    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:53.798349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.798307    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:53.799931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.799890    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:53.801009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.800978    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:53.801417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.801394    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:36:53.805605 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.805567    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:36:53.808502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.808457    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:36:53.808608 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.808570    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:53.809567 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.809536    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs" pod "5809f1df-91c9-484a-b1b2-25c4434c5a82" (UID: "5809f1df-91c9-484a-b1b2-25c4434c5a82"). InnerVolumeSpecName "kube-api-access-m82hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:36:53.898633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898597    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898633 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898628    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898640    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m82hs\" (UniqueName: \"kubernetes.io/projected/5809f1df-91c9-484a-b1b2-25c4434c5a82-kube-api-access-m82hs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898650    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898661    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898669    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898677    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898684    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5809f1df-91c9-484a-b1b2-25c4434c5a82-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:53.898834 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:53.898692    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5809f1df-91c9-484a-b1b2-25c4434c5a82-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:36:54.528292 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.528252    4453 generic.go:356] "Generic (PLEG): container finished" podID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerID="a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573" exitCode=0
Oct 30 12:36:54.528520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.528297    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerDied","Data":"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"}
Oct 30 12:36:54.528520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.528309    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"
Oct 30 12:36:54.528520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.528382    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m" event={"ID":"5809f1df-91c9-484a-b1b2-25c4434c5a82","Type":"ContainerDied","Data":"20033cec34ee1f40205b25d65b4779485f42314e85bde481c5655f4104598b75"}
Oct 30 12:36:54.528520 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.528417    4453 scope.go:122] "RemoveContainer" containerID="a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"
Oct 30 12:36:54.540572 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.540551    4453 scope.go:122] "RemoveContainer" containerID="b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"
Oct 30 12:36:54.550664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.550644    4453 scope.go:122] "RemoveContainer" containerID="a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"
Oct 30 12:36:54.550946 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:54.550919    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573\": container with ID starting with a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573 not found: ID does not exist" containerID="a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"
Oct 30 12:36:54.550995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.550958    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573"} err="failed to get container status \"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573\": rpc error: code = NotFound desc = could not find container \"a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573\": container with ID starting with a86dfdcd9a519864a930a188b89876615c0005cd1459414a8090e37b2f1de573 not found: ID does not exist"
Oct 30 12:36:54.550995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.550983    4453 scope.go:122] "RemoveContainer" containerID="b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"
Oct 30 12:36:54.551245 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:36:54.551225    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca\": container with ID starting with b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca not found: ID does not exist" containerID="b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"
Oct 30 12:36:54.551320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.551250    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca"} err="failed to get container status \"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca\": rpc error: code = NotFound desc = could not find container \"b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca\": container with ID starting with b77d300c4d55023326c568f1cc226b3f7604489bba5b192925b319ed08a376ca not found: ID does not exist"
Oct 30 12:36:54.591939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.591911    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:36:54.603239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.603203    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-31a6e3-openshift-default-77c678b749-jfj7m"]
Oct 30 12:36:54.912554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:54.912519    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" path="/var/lib/kubelet/pods/5809f1df-91c9-484a-b1b2-25c4434c5a82/volumes"
Oct 30 12:36:59.558661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:59.558577    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:36:59.591005 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:36:59.590975    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:37:04.559645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.559575    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" podUID="758b1d1b-3cda-4f5a-af48-453121857572" containerName="payload-processing" containerID="cri-o://3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b" gracePeriod=25
Oct 30 12:37:04.592034 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.591997    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing" containerID="cri-o://ca3b89908dab064384660469e16e5cbf817bf26909e62fb19bb3e57770cd9503" gracePeriod=25
Oct 30 12:37:04.846477 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.846450    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:37:04.943072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.943040    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\" (UniqueName: \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\") pod \"758b1d1b-3cda-4f5a-af48-453121857572\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") "
Oct 30 12:37:04.943239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.943217    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\") pod \"758b1d1b-3cda-4f5a-af48-453121857572\" (UID: \"758b1d1b-3cda-4f5a-af48-453121857572\") "
Oct 30 12:37:04.943653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.943614    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume" pod "758b1d1b-3cda-4f5a-af48-453121857572" (UID: "758b1d1b-3cda-4f5a-af48-453121857572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:04.945116 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:04.945096    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6" pod "758b1d1b-3cda-4f5a-af48-453121857572" (UID: "758b1d1b-3cda-4f5a-af48-453121857572"). InnerVolumeSpecName "kube-api-access-7dqp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:05.045189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.045156    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758b1d1b-3cda-4f5a-af48-453121857572-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:05.045189 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.045184    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dqp6\" (UniqueName: \"kubernetes.io/projected/758b1d1b-3cda-4f5a-af48-453121857572-kube-api-access-7dqp6\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:05.601972 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.601929    4453 generic.go:356] "Generic (PLEG): container finished" podID="758b1d1b-3cda-4f5a-af48-453121857572" containerID="3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b" exitCode=0
Oct 30 12:37:05.602485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.601981    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"
Oct 30 12:37:05.602485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.602019    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" event={"ID":"758b1d1b-3cda-4f5a-af48-453121857572","Type":"ContainerDied","Data":"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"}
Oct 30 12:37:05.602485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.602075    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk" event={"ID":"758b1d1b-3cda-4f5a-af48-453121857572","Type":"ContainerDied","Data":"f5367bc33e30aa32b63999eca82908c83c7f1f2800aa995fd64f4e474fa9fe64"}
Oct 30 12:37:05.602485 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.602104    4453 scope.go:122] "RemoveContainer" containerID="3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"
Oct 30 12:37:05.613109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.613088    4453 scope.go:122] "RemoveContainer" containerID="3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"
Oct 30 12:37:05.613384 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:37:05.613363    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b\": container with ID starting with 3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b not found: ID does not exist" containerID="3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"
Oct 30 12:37:05.613447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.613392    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b"} err="failed to get container status \"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b\": rpc error: code = NotFound desc = could not find container \"3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b\": container with ID starting with 3c7cfe6a292364eaea4fa05ec6e82868baf44c33e3ec56949d01fed46c4cea7b not found: ID does not exist"
Oct 30 12:37:05.654506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.654475    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:37:05.666478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:05.666448    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ffea51-55c47c4df4-kfzjk"]
Oct 30 12:37:06.912915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:06.912878    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758b1d1b-3cda-4f5a-af48-453121857572" path="/var/lib/kubelet/pods/758b1d1b-3cda-4f5a-af48-453121857572/volumes"
Oct 30 12:37:07.572030 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:37:07.571783    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:37:09.226191 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:09.226156    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 12:37:09.226680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:09.226412    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy" containerID="cri-o://ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820" gracePeriod=30
Oct 30 12:37:11.156643 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:11.156606    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:37:14.481541 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.481511    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 12:37:14.581162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581076    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581162 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581118    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581166    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581219    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581253    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581296    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581440 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581389    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:14.581701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581458    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581520    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:14.581701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581534    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581568    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\") pod \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\" (UID: \"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90\") "
Oct 30 12:37:14.581917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581730    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:14.581917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581828    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:14.581917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.581868    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:14.582296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.582271    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.582500 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.582483    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.582622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.582600    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.582756 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.582624    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.582756 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.582639    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.584035 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.584012    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:14.584133 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.584046    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "kube-api-access-78s57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:14.584198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.584167    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:37:14.584263 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.584220    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo" pod "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" (UID: "c750f8f8-84c6-4e9f-aaf7-b867db5d5c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:37:14.683821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.683787    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.683821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.683818    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.684020 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.683829    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78s57\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-kube-api-access-78s57\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.684020 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.683842    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:14.700838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.700801    4453 generic.go:356] "Generic (PLEG): container finished" podID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerID="ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820" exitCode=0
Oct 30 12:37:14.700976 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.700855    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"
Oct 30 12:37:14.700976 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.700876    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" event={"ID":"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90","Type":"ContainerDied","Data":"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"}
Oct 30 12:37:14.700976 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.700927    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n" event={"ID":"c750f8f8-84c6-4e9f-aaf7-b867db5d5c90","Type":"ContainerDied","Data":"6937e612735bf22ef2cf343362d1bb46304b281b9878179fc2aabd059be1ded0"}
Oct 30 12:37:14.700976 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.700948    4453 scope.go:122] "RemoveContainer" containerID="ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"
Oct 30 12:37:14.711789 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.711771    4453 scope.go:122] "RemoveContainer" containerID="ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"
Oct 30 12:37:14.712049 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:37:14.712030    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820\": container with ID starting with ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820 not found: ID does not exist" containerID="ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"
Oct 30 12:37:14.712100 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.712057    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820"} err="failed to get container status \"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820\": rpc error: code = NotFound desc = could not find container \"ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820\": container with ID starting with ae7dcd65dbada9babf214ebc7c230f3795c6ead31de608e1fbcd743e7670d820 not found: ID does not exist"
Oct 30 12:37:14.752121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.752091    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 12:37:14.758359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.758314    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-849669-openshift-default-694c9cbc97-9265n"]
Oct 30 12:37:14.912695 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:14.912666    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" path="/var/lib/kubelet/pods/c750f8f8-84c6-4e9f-aaf7-b867db5d5c90/volumes"
Oct 30 12:37:21.157467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:21.157430    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:37:24.771921 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:24.771878    4453 generic.go:356] "Generic (PLEG): container finished" podID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerID="ca3b89908dab064384660469e16e5cbf817bf26909e62fb19bb3e57770cd9503" exitCode=0
Oct 30 12:37:24.772323 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:24.771942    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" event={"ID":"73fd811c-b82c-4af1-96a7-0eff3b3aac2d","Type":"ContainerDied","Data":"ca3b89908dab064384660469e16e5cbf817bf26909e62fb19bb3e57770cd9503"}
Oct 30 12:37:24.915986 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:24.915956    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:37:25.048110 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.048072    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\") pod \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") "
Oct 30 12:37:25.048296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.048141    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\" (UniqueName: \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\") pod \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\" (UID: \"73fd811c-b82c-4af1-96a7-0eff3b3aac2d\") "
Oct 30 12:37:25.048515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.048486    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume" pod "73fd811c-b82c-4af1-96a7-0eff3b3aac2d" (UID: "73fd811c-b82c-4af1-96a7-0eff3b3aac2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:37:25.048723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.048704    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:25.050458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.050434    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577" pod "73fd811c-b82c-4af1-96a7-0eff3b3aac2d" (UID: "73fd811c-b82c-4af1-96a7-0eff3b3aac2d"). InnerVolumeSpecName "kube-api-access-86577". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:37:25.150195 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.150105    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86577\" (UniqueName: \"kubernetes.io/projected/73fd811c-b82c-4af1-96a7-0eff3b3aac2d-kube-api-access-86577\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:37:25.779634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.779603    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"
Oct 30 12:37:25.780137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.779637    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8" event={"ID":"73fd811c-b82c-4af1-96a7-0eff3b3aac2d","Type":"ContainerDied","Data":"47fe5e239a8d2e24d860c42aa7ca24ec9bcd2f1f5bcbcaab45916e2cf9765556"}
Oct 30 12:37:25.780137 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.779706    4453 scope.go:122] "RemoveContainer" containerID="ca3b89908dab064384660469e16e5cbf817bf26909e62fb19bb3e57770cd9503"
Oct 30 12:37:25.832641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.832608    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:37:25.843612 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:25.843582    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ffea51-6bd459d4c9-4qkc8"]
Oct 30 12:37:26.913582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:26.913548    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" path="/var/lib/kubelet/pods/73fd811c-b82c-4af1-96a7-0eff3b3aac2d/volumes"
Oct 30 12:37:34.601130 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:34.601091    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:37:37.571759 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:37:37.571720    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:37:39.602061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:39.602010    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing" containerID="cri-o://428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877" gracePeriod=25
Oct 30 12:37:46.495674 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:46.495632    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:37:56.495975 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:56.495930    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:37:59.887479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:37:59.887447    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:38:00.021467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.021425    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\" (UniqueName: \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\") pod \"1797d832-8456-42cf-83fb-d7fa3493498d\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") "
Oct 30 12:38:00.021672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.021593    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\") pod \"1797d832-8456-42cf-83fb-d7fa3493498d\" (UID: \"1797d832-8456-42cf-83fb-d7fa3493498d\") "
Oct 30 12:38:00.022012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.021975    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume" pod "1797d832-8456-42cf-83fb-d7fa3493498d" (UID: "1797d832-8456-42cf-83fb-d7fa3493498d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:00.022204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.022186    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1797d832-8456-42cf-83fb-d7fa3493498d-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:38:00.023801 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.023767    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2" pod "1797d832-8456-42cf-83fb-d7fa3493498d" (UID: "1797d832-8456-42cf-83fb-d7fa3493498d"). InnerVolumeSpecName "kube-api-access-pttb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:00.038165 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.038137    4453 generic.go:356] "Generic (PLEG): container finished" podID="1797d832-8456-42cf-83fb-d7fa3493498d" containerID="428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877" exitCode=0
Oct 30 12:38:00.038307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.038195    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"
Oct 30 12:38:00.038307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.038250    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" event={"ID":"1797d832-8456-42cf-83fb-d7fa3493498d","Type":"ContainerDied","Data":"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"}
Oct 30 12:38:00.038307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.038306    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl" event={"ID":"1797d832-8456-42cf-83fb-d7fa3493498d","Type":"ContainerDied","Data":"4901348a37dfb09ce0964509973f20028a1d2e434d813592ede894039dd201ca"}
Oct 30 12:38:00.038513 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.038360    4453 scope.go:122] "RemoveContainer" containerID="428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"
Oct 30 12:38:00.049753 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.049726    4453 scope.go:122] "RemoveContainer" containerID="428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"
Oct 30 12:38:00.050067 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:38:00.050035    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877\": container with ID starting with 428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877 not found: ID does not exist" containerID="428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"
Oct 30 12:38:00.050124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.050082    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877"} err="failed to get container status \"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877\": rpc error: code = NotFound desc = could not find container \"428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877\": container with ID starting with 428209564aa658c0086fa4cd316c1facc07aefb5cbaf39c844c34557314b6877 not found: ID does not exist"
Oct 30 12:38:00.093224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.093187    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:38:00.097744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.097715    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-59b8a9-5df69756b7-f2cbl"]
Oct 30 12:38:00.123354 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.123268    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttb2\" (UniqueName: \"kubernetes.io/projected/1797d832-8456-42cf-83fb-d7fa3493498d-kube-api-access-pttb2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:38:00.963355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:00.962634    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" path="/var/lib/kubelet/pods/1797d832-8456-42cf-83fb-d7fa3493498d/volumes"
Oct 30 12:38:07.571195 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:38:07.571154    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:38:32.162253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:32.162211    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:38:37.162735 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:37.162644    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing" containerID="cri-o://ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072" gracePeriod=25
Oct 30 12:38:37.572613 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:38:37.572466    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:38:42.266052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:42.266014    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:38:52.266286 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:52.266249    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:38:57.428837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.428775    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:38:57.463809 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.463773    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\") pod \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") "
Oct 30 12:38:57.464029 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.463985    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\" (UniqueName: \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\") pod \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\" (UID: \"7ba0dbf8-9300-41c3-867d-81aae4ccda53\") "
Oct 30 12:38:57.464130 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.464108    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume" pod "7ba0dbf8-9300-41c3-867d-81aae4ccda53" (UID: "7ba0dbf8-9300-41c3-867d-81aae4ccda53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:38:57.464374 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.464360    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba0dbf8-9300-41c3-867d-81aae4ccda53-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:38:57.466072 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.466047    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd" pod "7ba0dbf8-9300-41c3-867d-81aae4ccda53" (UID: "7ba0dbf8-9300-41c3-867d-81aae4ccda53"). InnerVolumeSpecName "kube-api-access-dxwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:38:57.496554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.496518    4453 generic.go:356] "Generic (PLEG): container finished" podID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerID="ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072" exitCode=0
Oct 30 12:38:57.496706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.496569    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"
Oct 30 12:38:57.496706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.496608    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" event={"ID":"7ba0dbf8-9300-41c3-867d-81aae4ccda53","Type":"ContainerDied","Data":"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"}
Oct 30 12:38:57.496706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.496680    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg" event={"ID":"7ba0dbf8-9300-41c3-867d-81aae4ccda53","Type":"ContainerDied","Data":"1a36f4a2f96f96308035b8be3d15e1184fd8d4f05dbce99c8c85826b61e140e8"}
Oct 30 12:38:57.496706 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.496696    4453 scope.go:122] "RemoveContainer" containerID="ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"
Oct 30 12:38:57.509387 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.509316    4453 scope.go:122] "RemoveContainer" containerID="ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"
Oct 30 12:38:57.509711 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:38:57.509678    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072\": container with ID starting with ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072 not found: ID does not exist" containerID="ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"
Oct 30 12:38:57.509857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.509718    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072"} err="failed to get container status \"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072\": rpc error: code = NotFound desc = could not find container \"ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072\": container with ID starting with ed818e12ddd2fa056f085a87445adbeb905867e1237cf375a29dafb8faa52072 not found: ID does not exist"
Oct 30 12:38:57.549556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.549519    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:38:57.556662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.556634    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b15ec3-745649cf87-896bg"]
Oct 30 12:38:57.565297 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:57.565271    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwhd\" (UniqueName: \"kubernetes.io/projected/7ba0dbf8-9300-41c3-867d-81aae4ccda53-kube-api-access-dxwhd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:38:58.913686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:38:58.913647    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" path="/var/lib/kubelet/pods/7ba0dbf8-9300-41c3-867d-81aae4ccda53/volumes"
Oct 30 12:39:04.470302 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.470261    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:39:04.470818 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.470508    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" podUID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" containerName="maas-api" containerID="cri-o://c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d" gracePeriod=30
Oct 30 12:39:04.825899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.825868    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:39:04.973955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.973910    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\" (UniqueName: \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\") pod \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") "
Oct 30 12:39:04.974538 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.974502    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\") pod \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") "
Oct 30 12:39:04.975777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.975751    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca" pod "af19fa44-557b-4b16-9d34-e39ecaf9ac08" (UID: "af19fa44-557b-4b16-9d34-e39ecaf9ac08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:39:04.975890 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.975815    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\") pod \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") "
Oct 30 12:39:04.976019 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.975994    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\") pod \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\" (UID: \"af19fa44-557b-4b16-9d34-e39ecaf9ac08\") "
Oct 30 12:39:04.978730 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.977938    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af19fa44-557b-4b16-9d34-e39ecaf9ac08-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:39:04.988576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:04.988528    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls" pod "af19fa44-557b-4b16-9d34-e39ecaf9ac08" (UID: "af19fa44-557b-4b16-9d34-e39ecaf9ac08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:39:05.007194 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.003759    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p" pod "af19fa44-557b-4b16-9d34-e39ecaf9ac08" (UID: "af19fa44-557b-4b16-9d34-e39ecaf9ac08"). InnerVolumeSpecName "kube-api-access-k725p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:39:05.007686 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.007634    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs" pod "af19fa44-557b-4b16-9d34-e39ecaf9ac08" (UID: "af19fa44-557b-4b16-9d34-e39ecaf9ac08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:39:05.079275 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.079194    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k725p\" (UniqueName: \"kubernetes.io/projected/af19fa44-557b-4b16-9d34-e39ecaf9ac08-kube-api-access-k725p\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:39:05.079275 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.079222    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:39:05.079275 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.079232    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af19fa44-557b-4b16-9d34-e39ecaf9ac08-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:39:05.577647 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.577613    4453 generic.go:356] "Generic (PLEG): container finished" podID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" containerID="c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d" exitCode=0
Oct 30 12:39:05.578014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.577660    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"
Oct 30 12:39:05.578014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.577698    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" event={"ID":"af19fa44-557b-4b16-9d34-e39ecaf9ac08","Type":"ContainerDied","Data":"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"}
Oct 30 12:39:05.578014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.577751    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5" event={"ID":"af19fa44-557b-4b16-9d34-e39ecaf9ac08","Type":"ContainerDied","Data":"3e2c89a9a3e95616397f4a62d7b6114583e343aea97ff38131d35c16551d8939"}
Oct 30 12:39:05.578014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.577774    4453 scope.go:122] "RemoveContainer" containerID="c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"
Oct 30 12:39:05.587799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.587783    4453 scope.go:122] "RemoveContainer" containerID="c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"
Oct 30 12:39:05.588046 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:39:05.588028    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d\": container with ID starting with c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d not found: ID does not exist" containerID="c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"
Oct 30 12:39:05.588109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.588053    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d"} err="failed to get container status \"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d\": rpc error: code = NotFound desc = could not find container \"c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d\": container with ID starting with c434ff6d403e0dca87b1a5208487d59c92488009c72971b1573e83f7fa12c21d not found: ID does not exist"
Oct 30 12:39:05.627967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.627941    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:39:05.639366 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:05.639322    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b15ec3-57bd848569-8ttn5"]
Oct 30 12:39:06.911936 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:39:06.911901    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" path="/var/lib/kubelet/pods/af19fa44-557b-4b16-9d34-e39ecaf9ac08/volumes"
Oct 30 12:39:07.571649 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:39:07.571610    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:39:37.652391 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:39:37.652081    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:40:07.571394 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:07.571344    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:40:11.605575 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:11.605539    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 12:40:11.618041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:11.618013    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 12:40:17.457851 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:17.457815    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 12:40:17.492004 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:17.491961    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 12:40:22.458636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:22.458591    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing" containerID="cri-o://8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0" gracePeriod=25
Oct 30 12:40:22.493220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:22.493185    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing" containerID="cri-o://0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf" gracePeriod=25
Oct 30 12:40:25.947582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:25.947541    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:25.948011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:25.947540    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:26.627609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.627557    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" containerID="cri-o://667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec" gracePeriod=45
Oct 30 12:40:26.632458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.632420    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": EOF"
Oct 30 12:40:26.638528 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.638498    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" containerID="cri-o://3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580" gracePeriod=45
Oct 30 12:40:26.683154 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.683108    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Oct 30 12:40:26.971718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.971687    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 12:40:26.974492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:26.974472    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 12:40:27.046041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046002    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.046257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046079    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.046257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046111    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.046257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046210    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\" (UniqueName: \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.046479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046282    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.046479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046365    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\" (UniqueName: \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.046479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046398    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046480    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046542    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046359    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046607    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\" (UniqueName: \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046392    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.046661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.046583    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.047747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.047186    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\") pod \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\" (UID: \"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1\") "
Oct 30 12:40:27.047747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.047269    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.047747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.047273    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\") pod \"1641683c-5de9-4297-87d5-75276e757516\" (UID: \"1641683c-5de9-4297-87d5-75276e757516\") "
Oct 30 12:40:27.047747 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.047495    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.048092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.048067    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.048168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.048099    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.048168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.048115    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.048168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.048130    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.048168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.048143    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.051893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.049269    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.051893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.050871    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "kube-api-access-6pzvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:27.052083 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.052019    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:27.052228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.052202    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.052804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.052776    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:40:27.054732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.054696    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs" pod "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" (UID: "1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:27.054978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.054951    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd" pod "1641683c-5de9-4297-87d5-75276e757516" (UID: "1641683c-5de9-4297-87d5-75276e757516"). InnerVolumeSpecName "kube-api-access-bxrkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:27.149854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149822    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzvf\" (UniqueName: \"kubernetes.io/projected/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-kube-api-access-6pzvf\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.149854 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149851    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.150062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149863    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrkd\" (UniqueName: \"kubernetes.io/projected/1641683c-5de9-4297-87d5-75276e757516-kube-api-access-bxrkd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.150062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149873    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.150062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149892    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1641683c-5de9-4297-87d5-75276e757516-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.150062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149899    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1641683c-5de9-4297-87d5-75276e757516-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.150062 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.149907    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:27.285998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.285911    4453 generic.go:356] "Generic (PLEG): container finished" podID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerID="3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580" exitCode=0
Oct 30 12:40:27.285998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.285972    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"
Oct 30 12:40:27.286233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.286000    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" event={"ID":"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1","Type":"ContainerDied","Data":"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"}
Oct 30 12:40:27.286233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.286045    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l" event={"ID":"1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1","Type":"ContainerDied","Data":"f61b3e65635f0f3f882c0e59e53b7bf2860aaf179560bef40c86c44c10947cb5"}
Oct 30 12:40:27.286233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.286071    4453 scope.go:122] "RemoveContainer" containerID="3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"
Oct 30 12:40:27.288491 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.288467    4453 generic.go:356] "Generic (PLEG): container finished" podID="1641683c-5de9-4297-87d5-75276e757516" containerID="667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec" exitCode=0
Oct 30 12:40:27.288890 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.288870    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"
Oct 30 12:40:27.289826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.289800    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" event={"ID":"1641683c-5de9-4297-87d5-75276e757516","Type":"ContainerDied","Data":"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"}
Oct 30 12:40:27.289942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.289843    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8" event={"ID":"1641683c-5de9-4297-87d5-75276e757516","Type":"ContainerDied","Data":"915abd029970e80fbeb53251988ed1675175030be0691178a3857e85afab078b"}
Oct 30 12:40:27.299653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.299618    4453 scope.go:122] "RemoveContainer" containerID="3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"
Oct 30 12:40:27.300553 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:27.300173    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580\": container with ID starting with 3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580 not found: ID does not exist" containerID="3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"
Oct 30 12:40:27.300553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.300210    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580"} err="failed to get container status \"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580\": rpc error: code = NotFound desc = could not find container \"3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580\": container with ID starting with 3a8d82d9b00f919890092e6dfefc7f852d71e1a3955d9a08026d19d2d812d580 not found: ID does not exist"
Oct 30 12:40:27.300553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.300234    4453 scope.go:122] "RemoveContainer" containerID="667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"
Oct 30 12:40:27.310446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.310421    4453 scope.go:122] "RemoveContainer" containerID="667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"
Oct 30 12:40:27.310735 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:27.310712    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec\": container with ID starting with 667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec not found: ID does not exist" containerID="667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"
Oct 30 12:40:27.310815 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.310745    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec"} err="failed to get container status \"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec\": rpc error: code = NotFound desc = could not find container \"667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec\": container with ID starting with 667ef8a2fa6e6b3c62c99fcef26e11fcfac6500cf019c4ba8a3d399241bb9aec not found: ID does not exist"
Oct 30 12:40:27.349048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.349011    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 12:40:27.353917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.353890    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/distinct-2-w3-4e261b-kserve-c9c6c6df-4cjw8"]
Oct 30 12:40:27.381624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.381595    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 12:40:27.384546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:27.384516    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4e261b/premium-w3-4e261b-kserve-7579bcf9ff-qjn2l"]
Oct 30 12:40:28.902858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:28.902824    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:40:28.903277 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:28.903047    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" podUID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" containerName="maas-api" containerID="cri-o://1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e" gracePeriod=30
Oct 30 12:40:28.915122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:28.915093    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1641683c-5de9-4297-87d5-75276e757516" path="/var/lib/kubelet/pods/1641683c-5de9-4297-87d5-75276e757516/volumes"
Oct 30 12:40:28.915771 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:28.915740    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" path="/var/lib/kubelet/pods/1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1/volumes"
Oct 30 12:40:29.020487 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.020457    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:40:29.053359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.053279    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:40:29.195956 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.195929    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:40:29.278495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.278450    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\") pod \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") "
Oct 30 12:40:29.278685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.278651    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\" (UniqueName: \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\") pod \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") "
Oct 30 12:40:29.278769 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.278751    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\") pod \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") "
Oct 30 12:40:29.278886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.278865    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\") pod \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\" (UID: \"9b15dee6-bff0-45cb-91f8-73cf81cccdd6\") "
Oct 30 12:40:29.279488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.279459    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca" pod "9b15dee6-bff0-45cb-91f8-73cf81cccdd6" (UID: "9b15dee6-bff0-45cb-91f8-73cf81cccdd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:29.279793 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.279768    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:29.288807 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.288784    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls" pod "9b15dee6-bff0-45cb-91f8-73cf81cccdd6" (UID: "9b15dee6-bff0-45cb-91f8-73cf81cccdd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:29.288915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.288807    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf" pod "9b15dee6-bff0-45cb-91f8-73cf81cccdd6" (UID: "9b15dee6-bff0-45cb-91f8-73cf81cccdd6"). InnerVolumeSpecName "kube-api-access-4j9qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:29.292060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.292034    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs" pod "9b15dee6-bff0-45cb-91f8-73cf81cccdd6" (UID: "9b15dee6-bff0-45cb-91f8-73cf81cccdd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:40:29.306927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.306891    4453 generic.go:356] "Generic (PLEG): container finished" podID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" containerID="1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e" exitCode=0
Oct 30 12:40:29.307042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.306937    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"
Oct 30 12:40:29.307042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.306962    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" event={"ID":"9b15dee6-bff0-45cb-91f8-73cf81cccdd6","Type":"ContainerDied","Data":"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"}
Oct 30 12:40:29.307042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.307012    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt" event={"ID":"9b15dee6-bff0-45cb-91f8-73cf81cccdd6","Type":"ContainerDied","Data":"c17740864f907eb7949109de8050e10c305054ac832edceaeef88c250e0f2503"}
Oct 30 12:40:29.307042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.307036    4453 scope.go:122] "RemoveContainer" containerID="1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"
Oct 30 12:40:29.317636 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.317618    4453 scope.go:122] "RemoveContainer" containerID="1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"
Oct 30 12:40:29.317911 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:29.317891    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e\": container with ID starting with 1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e not found: ID does not exist" containerID="1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"
Oct 30 12:40:29.317964 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.317921    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e"} err="failed to get container status \"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e\": rpc error: code = NotFound desc = could not find container \"1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e\": container with ID starting with 1585c05c7be044bebdb1193fc7a53a5bec42d47b7cd72634cacdc6b2dc6b2a5e not found: ID does not exist"
Oct 30 12:40:29.351712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.351687    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:40:29.355750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.355728    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-737f64-5df4d54998-cnlxt"]
Oct 30 12:40:29.384118 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.384084    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:29.384415 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.384392    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:29.384566 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:29.384544    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9qf\" (UniqueName: \"kubernetes.io/projected/9b15dee6-bff0-45cb-91f8-73cf81cccdd6-kube-api-access-4j9qf\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:30.913232 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:30.913199    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" path="/var/lib/kubelet/pods/9b15dee6-bff0-45cb-91f8-73cf81cccdd6/volumes"
Oct 30 12:40:34.021203 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.021162    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" podUID="59684744-248f-41fb-833b-e92effd276d0" containerName="payload-processing" containerID="cri-o://7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea" gracePeriod=25
Oct 30 12:40:34.060207 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.055906    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing" containerID="cri-o://5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f" gracePeriod=25
Oct 30 12:40:34.321190 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.321159    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:40:34.338919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.338888    4453 generic.go:356] "Generic (PLEG): container finished" podID="59684744-248f-41fb-833b-e92effd276d0" containerID="7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea" exitCode=0
Oct 30 12:40:34.339044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.338920    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" event={"ID":"59684744-248f-41fb-833b-e92effd276d0","Type":"ContainerDied","Data":"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"}
Oct 30 12:40:34.339044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.338975    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c" event={"ID":"59684744-248f-41fb-833b-e92effd276d0","Type":"ContainerDied","Data":"a8f60a3ffa0ae48569a44596a9acadc55fa61acb9c231f96936968ccdec67e60"}
Oct 30 12:40:34.339044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.338997    4453 scope.go:122] "RemoveContainer" containerID="7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"
Oct 30 12:40:34.339044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.338929    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"
Oct 30 12:40:34.348457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.348439    4453 scope.go:122] "RemoveContainer" containerID="7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"
Oct 30 12:40:34.348714 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:34.348692    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea\": container with ID starting with 7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea not found: ID does not exist" containerID="7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"
Oct 30 12:40:34.348787 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.348726    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea"} err="failed to get container status \"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea\": rpc error: code = NotFound desc = could not find container \"7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea\": container with ID starting with 7b85895d7ef9bf252c20748b87bd1a4e2e21a159966520f25f48c0dd07988cea not found: ID does not exist"
Oct 30 12:40:34.450009 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.449978    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\" (UniqueName: \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\") pod \"59684744-248f-41fb-833b-e92effd276d0\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") "
Oct 30 12:40:34.450208 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.450049    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\") pod \"59684744-248f-41fb-833b-e92effd276d0\" (UID: \"59684744-248f-41fb-833b-e92effd276d0\") "
Oct 30 12:40:34.450493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.450465    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume" pod "59684744-248f-41fb-833b-e92effd276d0" (UID: "59684744-248f-41fb-833b-e92effd276d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:34.452222 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.452196    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq" pod "59684744-248f-41fb-833b-e92effd276d0" (UID: "59684744-248f-41fb-833b-e92effd276d0"). InnerVolumeSpecName "kube-api-access-c9rbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:34.551868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.551796    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59684744-248f-41fb-833b-e92effd276d0-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:34.551868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.551823    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rbq\" (UniqueName: \"kubernetes.io/projected/59684744-248f-41fb-833b-e92effd276d0-kube-api-access-c9rbq\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:34.681493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.681462    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:40:34.690566 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.690529    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-737f64-fc5944d86-msl4c"]
Oct 30 12:40:34.912667 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:34.912635    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59684744-248f-41fb-833b-e92effd276d0" path="/var/lib/kubelet/pods/59684744-248f-41fb-833b-e92effd276d0/volumes"
Oct 30 12:40:35.948068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:35.948026    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:35.948496 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:35.948026    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:36.541886 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:36.541843    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:37.571214 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:37.571171    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:40:42.840094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.837773    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 12:40:42.850167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.850134    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 12:40:42.918153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.918121    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\" (UniqueName: \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\") pod \"4d146c4f-3052-4e60-8dbd-e44c4463decc\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") "
Oct 30 12:40:42.918431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.918411    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\" (UniqueName: \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\") pod \"987b63ab-2980-4e31-934a-b9c790783b19\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") "
Oct 30 12:40:42.918625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.918607    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\") pod \"987b63ab-2980-4e31-934a-b9c790783b19\" (UID: \"987b63ab-2980-4e31-934a-b9c790783b19\") "
Oct 30 12:40:42.918897 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.918877    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\") pod \"4d146c4f-3052-4e60-8dbd-e44c4463decc\" (UID: \"4d146c4f-3052-4e60-8dbd-e44c4463decc\") "
Oct 30 12:40:42.919766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.919743    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume" pod "4d146c4f-3052-4e60-8dbd-e44c4463decc" (UID: "4d146c4f-3052-4e60-8dbd-e44c4463decc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:42.920274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.920249    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume" pod "987b63ab-2980-4e31-934a-b9c790783b19" (UID: "987b63ab-2980-4e31-934a-b9c790783b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:42.921786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.921757    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms" pod "4d146c4f-3052-4e60-8dbd-e44c4463decc" (UID: "4d146c4f-3052-4e60-8dbd-e44c4463decc"). InnerVolumeSpecName "kube-api-access-8s7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:42.923046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:42.923022    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm" pod "987b63ab-2980-4e31-934a-b9c790783b19" (UID: "987b63ab-2980-4e31-934a-b9c790783b19"). InnerVolumeSpecName "kube-api-access-c8vpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:43.020298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.020229    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s7ms\" (UniqueName: \"kubernetes.io/projected/4d146c4f-3052-4e60-8dbd-e44c4463decc-kube-api-access-8s7ms\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.020298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.020254    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vpm\" (UniqueName: \"kubernetes.io/projected/987b63ab-2980-4e31-934a-b9c790783b19-kube-api-access-c8vpm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.020298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.020264    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987b63ab-2980-4e31-934a-b9c790783b19-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.020298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.020273    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d146c4f-3052-4e60-8dbd-e44c4463decc-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:43.409814 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.409776    4453 generic.go:356] "Generic (PLEG): container finished" podID="987b63ab-2980-4e31-934a-b9c790783b19" containerID="8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0" exitCode=0
Oct 30 12:40:43.410024 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.409820    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"
Oct 30 12:40:43.410024 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.409824    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" event={"ID":"987b63ab-2980-4e31-934a-b9c790783b19","Type":"ContainerDied","Data":"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"}
Oct 30 12:40:43.410024 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.409894    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx" event={"ID":"987b63ab-2980-4e31-934a-b9c790783b19","Type":"ContainerDied","Data":"8ac781d7edd7c28c876c44130af9a25385ac9d167f1b183b8694fa86dfad278b"}
Oct 30 12:40:43.410024 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.409908    4453 scope.go:122] "RemoveContainer" containerID="8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"
Oct 30 12:40:43.411439 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.411408    4453 generic.go:356] "Generic (PLEG): container finished" podID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerID="0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf" exitCode=0
Oct 30 12:40:43.411576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.411452    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"
Oct 30 12:40:43.411576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.411483    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" event={"ID":"4d146c4f-3052-4e60-8dbd-e44c4463decc","Type":"ContainerDied","Data":"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"}
Oct 30 12:40:43.411576 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.411527    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6" event={"ID":"4d146c4f-3052-4e60-8dbd-e44c4463decc","Type":"ContainerDied","Data":"4a34b2682ca1620ea06d88c6b9077ad2507cf9a22941c48949f11a7798387f98"}
Oct 30 12:40:43.421168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.421146    4453 scope.go:122] "RemoveContainer" containerID="8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"
Oct 30 12:40:43.421460 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:43.421434    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0\": container with ID starting with 8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0 not found: ID does not exist" containerID="8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"
Oct 30 12:40:43.421561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.421471    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0"} err="failed to get container status \"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0\": rpc error: code = NotFound desc = could not find container \"8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0\": container with ID starting with 8c3069ac4c647de34305991aeff7048b14a3db73342599f258b00fbcdaf5cdd0 not found: ID does not exist"
Oct 30 12:40:43.421561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.421496    4453 scope.go:122] "RemoveContainer" containerID="0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"
Oct 30 12:40:43.431259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.431244    4453 scope.go:122] "RemoveContainer" containerID="0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"
Oct 30 12:40:43.431532 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:43.431514    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf\": container with ID starting with 0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf not found: ID does not exist" containerID="0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"
Oct 30 12:40:43.431592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.431540    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf"} err="failed to get container status \"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf\": rpc error: code = NotFound desc = could not find container \"0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf\": container with ID starting with 0a831c8118248aa8a3428d05c576843713921b964b53d9599553b54aca24cabf not found: ID does not exist"
Oct 30 12:40:43.456912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.456889    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 12:40:43.484002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.483978    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4e261b-69f87b456c-nnbj6"]
Oct 30 12:40:43.519187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.519157    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 12:40:43.529592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:43.529568    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4e261b-865f9944fb-x7fwx"]
Oct 30 12:40:44.913287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:44.913250    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" path="/var/lib/kubelet/pods/4d146c4f-3052-4e60-8dbd-e44c4463decc/volumes"
Oct 30 12:40:44.914498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:44.914475    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987b63ab-2980-4e31-934a-b9c790783b19" path="/var/lib/kubelet/pods/987b63ab-2980-4e31-934a-b9c790783b19/volumes"
Oct 30 12:40:46.542855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:46.542805    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:40:50.876148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:50.876109    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:40:54.307124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.307097    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:40:54.395482 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.395449    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\" (UniqueName: \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\") pod \"54768444-c294-46d6-bd60-5b2b8c0d6157\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") "
Oct 30 12:40:54.395729 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.395498    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\") pod \"54768444-c294-46d6-bd60-5b2b8c0d6157\" (UID: \"54768444-c294-46d6-bd60-5b2b8c0d6157\") "
Oct 30 12:40:54.396274 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.396242    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume" pod "54768444-c294-46d6-bd60-5b2b8c0d6157" (UID: "54768444-c294-46d6-bd60-5b2b8c0d6157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:54.398135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.398106    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr" pod "54768444-c294-46d6-bd60-5b2b8c0d6157" (UID: "54768444-c294-46d6-bd60-5b2b8c0d6157"). InnerVolumeSpecName "kube-api-access-twlvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:54.508843 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.508804    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twlvr\" (UniqueName: \"kubernetes.io/projected/54768444-c294-46d6-bd60-5b2b8c0d6157-kube-api-access-twlvr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:54.509260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.509234    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54768444-c294-46d6-bd60-5b2b8c0d6157-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:54.531460 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.531419    4453 generic.go:356] "Generic (PLEG): container finished" podID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerID="5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f" exitCode=0
Oct 30 12:40:54.531653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.531582    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" event={"ID":"54768444-c294-46d6-bd60-5b2b8c0d6157","Type":"ContainerDied","Data":"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"}
Oct 30 12:40:54.531653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.531632    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw" event={"ID":"54768444-c294-46d6-bd60-5b2b8c0d6157","Type":"ContainerDied","Data":"d19de1a62ae22aea82afd38999790a7379d07a9c7fcc53cc1cde42fef41f153e"}
Oct 30 12:40:54.531773 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.531661    4453 scope.go:122] "RemoveContainer" containerID="5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"
Oct 30 12:40:54.531827 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.531818    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"
Oct 30 12:40:54.587680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.587569    4453 scope.go:122] "RemoveContainer" containerID="5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"
Oct 30 12:40:54.588090 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:54.588043    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f\": container with ID starting with 5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f not found: ID does not exist" containerID="5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"
Oct 30 12:40:54.588217 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.588104    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f"} err="failed to get container status \"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f\": rpc error: code = NotFound desc = could not find container \"5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f\": container with ID starting with 5a030835fbd335fe11f674e787951e1474033a27ad7c6276e5d5b1d4a170201f not found: ID does not exist"
Oct 30 12:40:54.624434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.624392    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:40:54.625725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.625700    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-737f64-697d98dc6c-7m4sw"]
Oct 30 12:40:54.923359 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:54.919366    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" path="/var/lib/kubelet/pods/54768444-c294-46d6-bd60-5b2b8c0d6157/volumes"
Oct 30 12:40:55.877124 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:55.877058    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" podUID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" containerName="payload-processing" containerID="cri-o://2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158" gracePeriod=25
Oct 30 12:40:56.163881 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.163847    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:40:56.243385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.242967    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\" (UniqueName: \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\") pod \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") "
Oct 30 12:40:56.243385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.243167    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\") pod \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\" (UID: \"613a10e5-5f00-45d0-bc5f-c9a575e90a50\") "
Oct 30 12:40:56.243728 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.243646    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume" pod "613a10e5-5f00-45d0-bc5f-c9a575e90a50" (UID: "613a10e5-5f00-45d0-bc5f-c9a575e90a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:40:56.248488 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.248452    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh" pod "613a10e5-5f00-45d0-bc5f-c9a575e90a50" (UID: "613a10e5-5f00-45d0-bc5f-c9a575e90a50"). InnerVolumeSpecName "kube-api-access-4hksh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:40:56.345248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.345203    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hksh\" (UniqueName: \"kubernetes.io/projected/613a10e5-5f00-45d0-bc5f-c9a575e90a50-kube-api-access-4hksh\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:56.345248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.345241    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613a10e5-5f00-45d0-bc5f-c9a575e90a50-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:40:56.550687 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.550589    4453 generic.go:356] "Generic (PLEG): container finished" podID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" containerID="2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158" exitCode=0
Oct 30 12:40:56.550687 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.550649    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"
Oct 30 12:40:56.550687 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.550653    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" event={"ID":"613a10e5-5f00-45d0-bc5f-c9a575e90a50","Type":"ContainerDied","Data":"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"}
Oct 30 12:40:56.550998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.550715    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv" event={"ID":"613a10e5-5f00-45d0-bc5f-c9a575e90a50","Type":"ContainerDied","Data":"a5b1a3f4973fe3e252483c909a95a26551a94ec9a2148eaa295ec2f255ca583f"}
Oct 30 12:40:56.550998 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.550746    4453 scope.go:122] "RemoveContainer" containerID="2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"
Oct 30 12:40:56.563042 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.563012    4453 scope.go:122] "RemoveContainer" containerID="2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"
Oct 30 12:40:56.563373 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:40:56.563347    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158\": container with ID starting with 2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158 not found: ID does not exist" containerID="2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"
Oct 30 12:40:56.563465 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.563385    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158"} err="failed to get container status \"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158\": rpc error: code = NotFound desc = could not find container \"2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158\": container with ID starting with 2fb50bb67d3311a8b8b1e44bb57cc29326561f5ce9e9dc7bfbc61231d20c6158 not found: ID does not exist"
Oct 30 12:40:56.594760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.594726    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:40:56.605453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.605412    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b8a9d-6b6dcd6988-mhqgv"]
Oct 30 12:40:56.915280 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:40:56.915240    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" path="/var/lib/kubelet/pods/613a10e5-5f00-45d0-bc5f-c9a575e90a50/volumes"
Oct 30 12:41:04.831779 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:04.831741    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 12:41:04.832279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:04.831948    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" containerName="maas-api" containerID="cri-o://b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19" gracePeriod=30
Oct 30 12:41:04.969725 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:04.969690    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.51:8443/health\": dial tcp 10.133.0.51:8443: connect: connection refused"
Oct 30 12:41:05.141392 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.111477    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.164151    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\") pod \"c23938af-54cd-4b54-b1b2-86057cc83721\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") "
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.164200    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\") pod \"c23938af-54cd-4b54-b1b2-86057cc83721\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") "
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.164264    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\" (UniqueName: \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\") pod \"c23938af-54cd-4b54-b1b2-86057cc83721\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") "
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.164293    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\") pod \"c23938af-54cd-4b54-b1b2-86057cc83721\" (UID: \"c23938af-54cd-4b54-b1b2-86057cc83721\") "
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.164913    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca" pod "c23938af-54cd-4b54-b1b2-86057cc83721" (UID: "c23938af-54cd-4b54-b1b2-86057cc83721"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:05.166270 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.165291    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c23938af-54cd-4b54-b1b2-86057cc83721-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:05.172480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.172438    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs" pod "c23938af-54cd-4b54-b1b2-86057cc83721" (UID: "c23938af-54cd-4b54-b1b2-86057cc83721"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:41:05.173010 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.172966    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7" pod "c23938af-54cd-4b54-b1b2-86057cc83721" (UID: "c23938af-54cd-4b54-b1b2-86057cc83721"). InnerVolumeSpecName "kube-api-access-52jr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:05.175135 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.175098    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls" pod "c23938af-54cd-4b54-b1b2-86057cc83721" (UID: "c23938af-54cd-4b54-b1b2-86057cc83721"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:41:05.266940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.266901    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:05.266940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.266932    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jr7\" (UniqueName: \"kubernetes.io/projected/c23938af-54cd-4b54-b1b2-86057cc83721-kube-api-access-52jr7\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:05.266940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.266947    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c23938af-54cd-4b54-b1b2-86057cc83721-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:05.644809 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.644772    4453 generic.go:356] "Generic (PLEG): container finished" podID="c23938af-54cd-4b54-b1b2-86057cc83721" containerID="b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19" exitCode=0
Oct 30 12:41:05.644985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.644827    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"
Oct 30 12:41:05.644985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.644850    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" event={"ID":"c23938af-54cd-4b54-b1b2-86057cc83721","Type":"ContainerDied","Data":"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"}
Oct 30 12:41:05.644985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.644920    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks" event={"ID":"c23938af-54cd-4b54-b1b2-86057cc83721","Type":"ContainerDied","Data":"043574b34e2c3bb559a7b9f132cbe51e5a4d74c37fc4377ecab2f9f2d5000568"}
Oct 30 12:41:05.644985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.644942    4453 scope.go:122] "RemoveContainer" containerID="b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"
Oct 30 12:41:05.655680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.655662    4453 scope.go:122] "RemoveContainer" containerID="b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"
Oct 30 12:41:05.655940 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:41:05.655923    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19\": container with ID starting with b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19 not found: ID does not exist" containerID="b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"
Oct 30 12:41:05.655994 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.655947    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19"} err="failed to get container status \"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19\": rpc error: code = NotFound desc = could not find container \"b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19\": container with ID starting with b13236f94fe769c97b7c9e1e5ba185a14826b33a774876e171fc7233276b5f19 not found: ID does not exist"
Oct 30 12:41:05.695574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.695549    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 12:41:05.705579 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:05.705556    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4e261b-68b954db47-4ghks"]
Oct 30 12:41:06.915680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:06.915648    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" path="/var/lib/kubelet/pods/c23938af-54cd-4b54-b1b2-86057cc83721/volumes"
Oct 30 12:41:07.570719 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:41:07.570676    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:41:09.507570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:09.507532    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:41:09.508294 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:09.508257    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy" containerID="cri-o://3517314312c04041ba3cbaed1e34ef6d72adcc78c65289c517de9aaa9e6278a0" gracePeriod=30
Oct 30 12:41:12.946163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946130    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:41:12.946711 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946698    4453 cpu_manager.go:414] "containerMap: removing container" podUID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" containerName="maas-api"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946712    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" containerName="maas-api"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946728    4453 cpu_manager.go:414] "containerMap: removing container" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946734    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946741    4453 cpu_manager.go:414] "containerMap: removing container" podUID="4362bb18-c55b-4326-8208-90b80ab4571f" containerName="maas-api"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946746    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4362bb18-c55b-4326-8208-90b80ab4571f" containerName="maas-api"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946753    4453 cpu_manager.go:414] "containerMap: removing container" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946757    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946764    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:41:12.946770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946768    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5f06ab-42af-4d39-9776-f4f7d3009c2c" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946776    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946781    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946792    4453 cpu_manager.go:414] "containerMap: removing container" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946796    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946804    4453 cpu_manager.go:414] "containerMap: removing container" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946809    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946817    4453 cpu_manager.go:414] "containerMap: removing container" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946821    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946829    4453 cpu_manager.go:414] "containerMap: removing container" podUID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946833    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946843    4453 cpu_manager.go:414] "containerMap: removing container" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946847    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999feecb-154b-4d75-85fd-08ea8fc965e8" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946855    4453 cpu_manager.go:414] "containerMap: removing container" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946859    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946867    4453 cpu_manager.go:414] "containerMap: removing container" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946871    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946878    4453 cpu_manager.go:414] "containerMap: removing container" podUID="59684744-248f-41fb-833b-e92effd276d0" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946882    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59684744-248f-41fb-833b-e92effd276d0" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946889    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946893    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946904    4453 cpu_manager.go:414] "containerMap: removing container" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946909    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946916    4453 cpu_manager.go:414] "containerMap: removing container" podUID="758b1d1b-3cda-4f5a-af48-453121857572" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946921    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758b1d1b-3cda-4f5a-af48-453121857572" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946930    4453 cpu_manager.go:414] "containerMap: removing container" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946935    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b3d23f-5af0-418b-9542-6d555273f0a5" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946942    4453 cpu_manager.go:414] "containerMap: removing container" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946947    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946954    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946959    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946967    4453 cpu_manager.go:414] "containerMap: removing container" podUID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946971    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" containerName="maas-api"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946977    4453 cpu_manager.go:414] "containerMap: removing container" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946981    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946993    4453 cpu_manager.go:414] "containerMap: removing container" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.946999    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947006    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947011    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947017    4453 cpu_manager.go:414] "containerMap: removing container" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947021    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947041    4453 cpu_manager.go:414] "containerMap: removing container" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947046    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ed9bc8-5a69-44d2-87e9-6f3b15be19d9" containerName="cleanup"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947054    4453 cpu_manager.go:414] "containerMap: removing container" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947059    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947068    4453 cpu_manager.go:414] "containerMap: removing container" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947073    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947079    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947085    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947092    4453 cpu_manager.go:414] "containerMap: removing container" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing"
Oct 30 12:41:12.947079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947097    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947108    4453 cpu_manager.go:414] "containerMap: removing container" podUID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947113    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947120    4453 cpu_manager.go:414] "containerMap: removing container" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947124    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947132    4453 cpu_manager.go:414] "containerMap: removing container" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947137    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947211    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23938af-54cd-4b54-b1b2-86057cc83721" containerName="maas-api"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947219    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="c750f8f8-84c6-4e9f-aaf7-b867db5d5c90" containerName="istio-proxy"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947226    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="59684744-248f-41fb-833b-e92effd276d0" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947233    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1797d832-8456-42cf-83fb-d7fa3493498d" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947240    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba0dbf8-9300-41c3-867d-81aae4ccda53" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947246    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="758b1d1b-3cda-4f5a-af48-453121857572" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947251    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="54768444-c294-46d6-bd60-5b2b8c0d6157" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947257    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a23799d-4c42-4f57-be2a-21c1067e6e2e" containerName="maas-api"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947263    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="af19fa44-557b-4b16-9d34-e39ecaf9ac08" containerName="maas-api"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947268    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a1c38-ecee-484c-bf0a-52dc4783379a" containerName="cleanup"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947275    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="872e9968-a3f9-4637-ac74-62eee04586d3" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947280    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec590f6-bd3d-4f76-8ff1-b7cac7cc1b0a" containerName="cleanup"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947286    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6a332d-96d7-4c32-9c7f-6212ff2a4c4b" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947293    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8443e82-cb7c-4042-a03f-0e51d8160358" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947298    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947304    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="987b63ab-2980-4e31-934a-b9c790783b19" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947310    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1acacdad-31aa-4fdc-aea1-83197367f134" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947314    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4fb54c-b6ac-48c3-addd-4378b0524b77" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947319    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cfa747-2078-4af2-97c2-0d3e02a8e8ed" containerName="cleanup"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947325    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="828dc1ab-5579-4734-b6ae-3cd75432c9dd" containerName="payload-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947343    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d146c4f-3052-4e60-8dbd-e44c4463decc" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947348    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1641683c-5de9-4297-87d5-75276e757516" containerName="main"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947354    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b15dee6-bff0-45cb-91f8-73cf81cccdd6" containerName="maas-api"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947360    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5dad6e-ec23-4f28-a550-cd49b2f7bc05" containerName="cleanup"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947367    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947372    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="4362bb18-c55b-4326-8208-90b80ab4571f" containerName="maas-api"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947378    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2931ba8-16df-4903-bfa9-8cf801e80efe" containerName="istio-proxy"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947384    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae6a366-7df5-47d6-84ef-e8a24f6d8ad1" containerName="main"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947391    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947395    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fd811c-b82c-4af1-96a7-0eff3b3aac2d" containerName="payload-pre-processing"
Oct 30 12:41:12.948662 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.947400    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="613a10e5-5f00-45d0-bc5f-c9a575e90a50" containerName="payload-processing"
Oct 30 12:41:12.950872 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.950724    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:12.973079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.973055    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2c719a\"/\"default-dockercfg-vwrrj\""
Oct 30 12:41:12.973383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.973317    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-2c719a\"/\"rate-test-model-2c719a-kserve-self-signed-certs\""
Oct 30 12:41:12.977318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:12.977292    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:41:13.050727 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.050686    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2c719a\"/\"kube-root-ca.crt\""
Oct 30 12:41:13.050727 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.050731    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-2c719a\"/\"openshift-service-ca.crt\""
Oct 30 12:41:13.068535 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068508    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.068670 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068644    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.068731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068690    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.068808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068791    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.068853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068830    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.068904 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.068867    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7bf\" (UniqueName: \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.169682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169651    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.169860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169699    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7bf\" (UniqueName: \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.169860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169765    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.169860 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169847    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.170052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169894    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.170052 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.169975    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.170322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.170299    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.170480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.170380    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.170480 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.170467    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.172225 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.172194    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.172654 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.172617    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.190483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.190455    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7bf\" (UniqueName: \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\") pod \"rate-test-model-2c719a-kserve-657bd586df-hp7kf\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") " pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.281466 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.272347    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:13.633038 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.633011    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:41:13.634245 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:41:13.634218    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7b7ce4_d5ed_4065_a6af_a5de8502d541.slice/crio-c8300a17c4961824d2f85250cd673d00d2de59d96c28cee9ee4abc77fcaca203 WatchSource:0}: Error finding container c8300a17c4961824d2f85250cd673d00d2de59d96c28cee9ee4abc77fcaca203: Status 404 returned error can't find the container with id c8300a17c4961824d2f85250cd673d00d2de59d96c28cee9ee4abc77fcaca203
Oct 30 12:41:13.741454 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.741401    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" event={"ID":"9f7b7ce4-d5ed-4065-a6af-a5de8502d541","Type":"ContainerStarted","Data":"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"}
Oct 30 12:41:13.741607 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.741462    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" event={"ID":"9f7b7ce4-d5ed-4065-a6af-a5de8502d541","Type":"ContainerStarted","Data":"c8300a17c4961824d2f85250cd673d00d2de59d96c28cee9ee4abc77fcaca203"}
Oct 30 12:41:13.778609 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:13.778556    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" podStartSLOduration=1.7785416280000002 podStartE2EDuration="1.778541628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:41:13.776981529 +0000 UTC m=+4272.997266438" watchObservedRunningTime="2026-10-30 12:41:13.778541628 +0000 UTC m=+4272.998826538"
Oct 30 12:41:14.278060 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.278019    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:14.456882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.447473    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 12:41:14.456882 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.447742    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy" containerID="cri-o://31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38" gracePeriod=30
Oct 30 12:41:14.748987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.748954    4453 generic.go:356] "Generic (PLEG): container finished" podID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerID="3517314312c04041ba3cbaed1e34ef6d72adcc78c65289c517de9aaa9e6278a0" exitCode=0
Oct 30 12:41:14.749164 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.749031    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" event={"ID":"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4","Type":"ContainerDied","Data":"3517314312c04041ba3cbaed1e34ef6d72adcc78c65289c517de9aaa9e6278a0"}
Oct 30 12:41:14.792197 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.792142    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:41:14.916987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.916961    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917000    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917043    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917063    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917132 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917081    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917140    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917231    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917265    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917369    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\") pod \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\" (UID: \"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4\") "
Oct 30 12:41:14.917622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917396    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:14.917622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917492    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:14.917622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917504    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:14.917622 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917586    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:14.917837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917644    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:14.918777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.917938    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.918777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.918549    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.918777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.918567    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.918777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.918583    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.918777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.918608    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:14.920483 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.919860    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:41:14.920597 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.920570    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:14.930428 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.930399    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:14.931161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:14.931122    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj" pod "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" (UID: "5778e7e4-78ba-48ce-b019-ca52cd3c0aa4"). InnerVolumeSpecName "kube-api-access-xfnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:15.019924 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.019900    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:15.020071 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.020053    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:15.020131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.020076    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:15.020131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.020089    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnjj\" (UniqueName: \"kubernetes.io/projected/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4-kube-api-access-xfnjj\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:15.757479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.757446    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"
Oct 30 12:41:15.757479 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.757454    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv" event={"ID":"5778e7e4-78ba-48ce-b019-ca52cd3c0aa4","Type":"ContainerDied","Data":"c9050f0a4d37d64f02d07a00511f0ef75354abc2bae7230bb2414b2b384f093d"}
Oct 30 12:41:15.757989 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.757549    4453 scope.go:122] "RemoveContainer" containerID="3517314312c04041ba3cbaed1e34ef6d72adcc78c65289c517de9aaa9e6278a0"
Oct 30 12:41:15.811962 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.811932    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:41:15.821546 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:15.821515    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b8a9d-openshift-default-5bffc5fb8f-vngwv"]
Oct 30 12:41:16.950481 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:16.949232    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" path="/var/lib/kubelet/pods/5778e7e4-78ba-48ce-b019-ca52cd3c0aa4/volumes"
Oct 30 12:41:19.720690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.720631    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 12:41:19.792120 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.792080    4453 generic.go:356] "Generic (PLEG): container finished" podID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerID="31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38" exitCode=0
Oct 30 12:41:19.792266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.792185    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"
Oct 30 12:41:19.792266 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.792220    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" event={"ID":"c9483385-92b9-4b99-8f17-9b554199e2cf","Type":"ContainerDied","Data":"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"}
Oct 30 12:41:19.792376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.792288    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw" event={"ID":"c9483385-92b9-4b99-8f17-9b554199e2cf","Type":"ContainerDied","Data":"d98c45db7f9e1e70a599dd2277f50e0dfe5ac2af385dff7d70e62820ef8de1dd"}
Oct 30 12:41:19.792376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.792318    4453 scope.go:122] "RemoveContainer" containerID="31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"
Oct 30 12:41:19.803391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.803367    4453 scope.go:122] "RemoveContainer" containerID="31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"
Oct 30 12:41:19.803723 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:41:19.803701    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38\": container with ID starting with 31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38 not found: ID does not exist" containerID="31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"
Oct 30 12:41:19.803832 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.803734    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38"} err="failed to get container status \"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38\": rpc error: code = NotFound desc = could not find container \"31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38\": container with ID starting with 31c42b524c04890e2075f7a30f1fa7a64b6a11857acf828f0b4548dcdea3aa38 not found: ID does not exist"
Oct 30 12:41:19.826067 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826044    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826093    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826149    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826191    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826248    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826309 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826286    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826364    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826423    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826516 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826453    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\") pod \"c9483385-92b9-4b99-8f17-9b554199e2cf\" (UID: \"c9483385-92b9-4b99-8f17-9b554199e2cf\") "
Oct 30 12:41:19.826676 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826545    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:19.826676 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826656    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:19.826788 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826756    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:19.827061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.826905    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:19.827196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827185    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:41:19.827553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827474    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9483385-92b9-4b99-8f17-9b554199e2cf-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.827553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827498    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.827553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827510    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.827553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827523    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.827553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.827535    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.828837 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.828812    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "kube-api-access-tlbp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:19.828990 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.828972    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:41:19.829047 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.829006    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:41:19.829260 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.829235    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy" pod "c9483385-92b9-4b99-8f17-9b554199e2cf" (UID: "c9483385-92b9-4b99-8f17-9b554199e2cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:41:19.928848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.928811    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.928848 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.928843    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbp8\" (UniqueName: \"kubernetes.io/projected/c9483385-92b9-4b99-8f17-9b554199e2cf-kube-api-access-tlbp8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.929059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.928857    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:19.929059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:19.928870    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9483385-92b9-4b99-8f17-9b554199e2cf-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:41:20.135088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:20.135050    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 12:41:20.144155 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:20.144126    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4e261b-openshift-default-6bf9798c75-r2qhw"]
Oct 30 12:41:20.913138 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:20.913107    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" path="/var/lib/kubelet/pods/c9483385-92b9-4b99-8f17-9b554199e2cf/volumes"
Oct 30 12:41:23.277545 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:23.277512    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:23.290573 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:23.290547    4453 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:23.830726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:41:23.830698    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:41:37.570299 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:41:37.570263    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:42:07.571204 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:42:07.571165    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:42:36.514045 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:36.514009    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 12:42:36.544967 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:36.544939    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 12:42:37.571263 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:42:37.571224    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:42:41.514572 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:41.514533    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing" containerID="cri-o://57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c" gracePeriod=25
Oct 30 12:42:41.546051 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:41.546010    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing" containerID="cri-o://4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a" gracePeriod=25
Oct 30 12:42:47.022953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:47.022856    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:42:47.023400 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:47.022867    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:42:57.022176 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:57.022135    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:42:57.023264 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:42:57.023237    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:43:01.814869 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.814841    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 12:43:01.838594 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.838569    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\") pod \"cac8816f-b9fd-4e11-99df-3314b405dcb7\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") "
Oct 30 12:43:01.838720 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.838708    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\" (UniqueName: \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\") pod \"cac8816f-b9fd-4e11-99df-3314b405dcb7\" (UID: \"cac8816f-b9fd-4e11-99df-3314b405dcb7\") "
Oct 30 12:43:01.839095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.838933    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume" pod "cac8816f-b9fd-4e11-99df-3314b405dcb7" (UID: "cac8816f-b9fd-4e11-99df-3314b405dcb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:01.839095 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.839074    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac8816f-b9fd-4e11-99df-3314b405dcb7-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:01.840802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.840782    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd" pod "cac8816f-b9fd-4e11-99df-3314b405dcb7" (UID: "cac8816f-b9fd-4e11-99df-3314b405dcb7"). InnerVolumeSpecName "kube-api-access-7b9bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:01.845971 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.845951    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 12:43:01.940918 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.940878    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\") pod \"fb44dea0-a97c-4fc7-860d-60d44d72d724\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") "
Oct 30 12:43:01.941115 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.940932    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\") pod \"fb44dea0-a97c-4fc7-860d-60d44d72d724\" (UID: \"fb44dea0-a97c-4fc7-860d-60d44d72d724\") "
Oct 30 12:43:01.941973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.941667    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9bd\" (UniqueName: \"kubernetes.io/projected/cac8816f-b9fd-4e11-99df-3314b405dcb7-kube-api-access-7b9bd\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:01.941973 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.941892    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume" pod "fb44dea0-a97c-4fc7-860d-60d44d72d724" (UID: "fb44dea0-a97c-4fc7-860d-60d44d72d724"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:01.965919 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:01.965867    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk" pod "fb44dea0-a97c-4fc7-860d-60d44d72d724" (UID: "fb44dea0-a97c-4fc7-860d-60d44d72d724"). InnerVolumeSpecName "kube-api-access-zn5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:02.043168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.043084    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5gk\" (UniqueName: \"kubernetes.io/projected/fb44dea0-a97c-4fc7-860d-60d44d72d724-kube-api-access-zn5gk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:02.043168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.043123    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb44dea0-a97c-4fc7-860d-60d44d72d724-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:02.454608 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.454574    4453 generic.go:356] "Generic (PLEG): container finished" podID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerID="4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a" exitCode=0
Oct 30 12:43:02.454808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.454618    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"
Oct 30 12:43:02.454808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.454640    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" event={"ID":"fb44dea0-a97c-4fc7-860d-60d44d72d724","Type":"ContainerDied","Data":"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"}
Oct 30 12:43:02.454808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.454694    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql" event={"ID":"fb44dea0-a97c-4fc7-860d-60d44d72d724","Type":"ContainerDied","Data":"f5818ce47c1e28d22578f5079ae285510dba88296d9678fab20a7c8fe91caecf"}
Oct 30 12:43:02.454808 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.454724    4453 scope.go:122] "RemoveContainer" containerID="4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"
Oct 30 12:43:02.456498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.456473    4453 generic.go:356] "Generic (PLEG): container finished" podID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerID="57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c" exitCode=0
Oct 30 12:43:02.456603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.456518    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"
Oct 30 12:43:02.456603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.456545    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" event={"ID":"cac8816f-b9fd-4e11-99df-3314b405dcb7","Type":"ContainerDied","Data":"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"}
Oct 30 12:43:02.456603 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.456585    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59" event={"ID":"cac8816f-b9fd-4e11-99df-3314b405dcb7","Type":"ContainerDied","Data":"4750e6e052e878881e21d13767565fbedb3980e6c3db947d263f877f8055393f"}
Oct 30 12:43:02.465408 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.465390    4453 scope.go:122] "RemoveContainer" containerID="4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"
Oct 30 12:43:02.465699 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:02.465681    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a\": container with ID starting with 4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a not found: ID does not exist" containerID="4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"
Oct 30 12:43:02.465745 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.465707    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a"} err="failed to get container status \"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a\": rpc error: code = NotFound desc = could not find container \"4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a\": container with ID starting with 4825cfcfeef831bc1e6adde68eac135a4d6c4260ad15125cdc90aa125bc2c68a not found: ID does not exist"
Oct 30 12:43:02.465745 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.465725    4453 scope.go:122] "RemoveContainer" containerID="57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"
Oct 30 12:43:02.474253 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.474237    4453 scope.go:122] "RemoveContainer" containerID="57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"
Oct 30 12:43:02.474519 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:02.474497    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c\": container with ID starting with 57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c not found: ID does not exist" containerID="57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"
Oct 30 12:43:02.474589 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.474528    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c"} err="failed to get container status \"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c\": rpc error: code = NotFound desc = could not find container \"57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c\": container with ID starting with 57dc55b3519afd757d702448b9767b72076912053f91d146290faffbe2d4583c not found: ID does not exist"
Oct 30 12:43:02.500136 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.500112    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 12:43:02.509790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.509766    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-849669-587b744455-f58ql"]
Oct 30 12:43:02.538992 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.538970    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 12:43:02.545885 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.545855    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-849669-f9698874b-mvl59"]
Oct 30 12:43:02.911313 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.911276    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" path="/var/lib/kubelet/pods/cac8816f-b9fd-4e11-99df-3314b405dcb7/volumes"
Oct 30 12:43:02.911739 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:02.911723    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" path="/var/lib/kubelet/pods/fb44dea0-a97c-4fc7-860d-60d44d72d724/volumes"
Oct 30 12:43:07.382928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.382897    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7377-edefe4"]
Oct 30 12:43:07.383498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383464    4453 cpu_manager.go:414] "containerMap: removing container" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing"
Oct 30 12:43:07.383498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383478    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing"
Oct 30 12:43:07.383498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383494    4453 cpu_manager.go:414] "containerMap: removing container" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy"
Oct 30 12:43:07.383498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383499    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383514    4453 cpu_manager.go:414] "containerMap: removing container" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383522    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9525f0-ff85-42d9-a124-e7c76c4d121f" containerName="istio-proxy"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383549    4453 cpu_manager.go:414] "containerMap: removing container" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383554    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383564    4453 cpu_manager.go:414] "containerMap: removing container" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383569    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383652    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac8816f-b9fd-4e11-99df-3314b405dcb7" containerName="payload-processing"
Oct 30 12:43:07.383665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383659    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="5778e7e4-78ba-48ce-b019-ca52cd3c0aa4" containerName="istio-proxy"
Oct 30 12:43:07.383985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383667    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb44dea0-a97c-4fc7-860d-60d44d72d724" containerName="payload-pre-processing"
Oct 30 12:43:07.383985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383677    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9483385-92b9-4b99-8f17-9b554199e2cf" containerName="istio-proxy"
Oct 30 12:43:07.383985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.383683    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="5809f1df-91c9-484a-b1b2-25c4434c5a82" containerName="istio-proxy"
Oct 30 12:43:07.388615 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.388599    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:07.401153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.401130    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-czl4d\""
Oct 30 12:43:07.404003 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.403977    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-edefe4"]
Oct 30 12:43:07.426786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.426756    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268zk\" (UniqueName: \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\") pod \"test-curl-7377-edefe4\" (UID: \"d19f7b61-1de2-42f5-9001-4e7e263fa520\") " pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:07.528303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.528265    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268zk\" (UniqueName: \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\") pod \"test-curl-7377-edefe4\" (UID: \"d19f7b61-1de2-42f5-9001-4e7e263fa520\") " pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:07.548498 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.548470    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-268zk\" (UniqueName: \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\") pod \"test-curl-7377-edefe4\" (UID: \"d19f7b61-1de2-42f5-9001-4e7e263fa520\") " pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:07.570805 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:07.570768    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:43:07.700699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:07.700628    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:08.002338 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:08.002303    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7377-edefe4"]
Oct 30 12:43:08.004390 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:43:08.004362    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19f7b61_1de2_42f5_9001_4e7e263fa520.slice/crio-b621515b39dfb9f8184275220b85e78af5cc3db5820a2d07f9e12d06bd8d89ed WatchSource:0}: Error finding container b621515b39dfb9f8184275220b85e78af5cc3db5820a2d07f9e12d06bd8d89ed: Status 404 returned error can't find the container with id b621515b39dfb9f8184275220b85e78af5cc3db5820a2d07f9e12d06bd8d89ed
Oct 30 12:43:08.492538 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:08.492502    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-edefe4" event={"ID":"d19f7b61-1de2-42f5-9001-4e7e263fa520","Type":"ContainerStarted","Data":"e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055"}
Oct 30 12:43:08.492538 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:08.492542    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7377-edefe4" event={"ID":"d19f7b61-1de2-42f5-9001-4e7e263fa520","Type":"ContainerStarted","Data":"b621515b39dfb9f8184275220b85e78af5cc3db5820a2d07f9e12d06bd8d89ed"}
Oct 30 12:43:08.528065 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:08.528014    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7377-edefe4" podStartSLOduration=1.528000118 podStartE2EDuration="1.528000118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:43:08.526073067 +0000 UTC m=+4387.746357977" watchObservedRunningTime="2026-10-30 12:43:08.528000118 +0000 UTC m=+4387.748285032"
Oct 30 12:43:09.529445 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:09.529411    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7377-edefe4"]
Oct 30 12:43:09.534224 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:09.534191    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7377-edefe4"]
Oct 30 12:43:10.504778 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:10.504732    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7377-edefe4" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" containerName="test-curl-7377-edefe4" containerID="cri-o://e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055" gracePeriod=2
Oct 30 12:43:10.515944 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:10.515912    4453 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-edefe4\" is forbidden: User \"system:node:ip-10-0-143-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-18.ec2.internal' and this object" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:10.918045 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:10.918005    4453 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7377-edefe4\" is forbidden: User \"system:node:ip-10-0-143-18.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-18.ec2.internal' and this object" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:12.710252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:12.710227    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:12.806582 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:12.806512    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\" (UniqueName: \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\") pod \"d19f7b61-1de2-42f5-9001-4e7e263fa520\" (UID: \"d19f7b61-1de2-42f5-9001-4e7e263fa520\") "
Oct 30 12:43:12.808506 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:12.808480    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk" pod "d19f7b61-1de2-42f5-9001-4e7e263fa520" (UID: "d19f7b61-1de2-42f5-9001-4e7e263fa520"). InnerVolumeSpecName "kube-api-access-268zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:12.908229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:12.908204    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268zk\" (UniqueName: \"kubernetes.io/projected/d19f7b61-1de2-42f5-9001-4e7e263fa520-kube-api-access-268zk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:12.911257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:12.911228    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" path="/var/lib/kubelet/pods/d19f7b61-1de2-42f5-9001-4e7e263fa520/volumes"
Oct 30 12:43:13.523774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:13.523741    4453 generic.go:356] "Generic (PLEG): container finished" podID="d19f7b61-1de2-42f5-9001-4e7e263fa520" containerID="e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055" exitCode=137
Oct 30 12:43:13.523952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:13.523785    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7377-edefe4"
Oct 30 12:43:13.523952 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:13.523844    4453 scope.go:122] "RemoveContainer" containerID="e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055"
Oct 30 12:43:13.533186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:13.533167    4453 scope.go:122] "RemoveContainer" containerID="e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055"
Oct 30 12:43:13.533482 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:13.533462    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055\": container with ID starting with e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055 not found: ID does not exist" containerID="e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055"
Oct 30 12:43:13.533522 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:13.533492    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055"} err="failed to get container status \"e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055\": rpc error: code = NotFound desc = could not find container \"e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055\": container with ID starting with e52b81b31c362fe09753670d21d36b54c71b5f67aadc20a9845ff86c35672055 not found: ID does not exist"
Oct 30 12:43:30.138404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.138366    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 12:43:30.138857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.138587    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" podUID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" containerName="maas-api" containerID="cri-o://f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9" gracePeriod=30
Oct 30 12:43:30.398216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.398160    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 12:43:30.538086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.538053    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\") pod \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") "
Oct 30 12:43:30.538287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.538130    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\") pod \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") "
Oct 30 12:43:30.538287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.538254    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\" (UniqueName: \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\") pod \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") "
Oct 30 12:43:30.538444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.538297    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\") pod \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\" (UID: \"69f08044-0f0d-47ce-8640-b4593e1ee5c1\") "
Oct 30 12:43:30.538815 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.538782    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca" pod "69f08044-0f0d-47ce-8640-b4593e1ee5c1" (UID: "69f08044-0f0d-47ce-8640-b4593e1ee5c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:30.540206 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.540172    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs" pod "69f08044-0f0d-47ce-8640-b4593e1ee5c1" (UID: "69f08044-0f0d-47ce-8640-b4593e1ee5c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:30.540299 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.540281    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls" pod "69f08044-0f0d-47ce-8640-b4593e1ee5c1" (UID: "69f08044-0f0d-47ce-8640-b4593e1ee5c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:43:30.540385 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.540309    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx" pod "69f08044-0f0d-47ce-8640-b4593e1ee5c1" (UID: "69f08044-0f0d-47ce-8640-b4593e1ee5c1"). InnerVolumeSpecName "kube-api-access-x56wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:30.633742 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.633704    4453 generic.go:356] "Generic (PLEG): container finished" podID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" containerID="f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9" exitCode=0
Oct 30 12:43:30.633935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.633910    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" event={"ID":"69f08044-0f0d-47ce-8640-b4593e1ee5c1","Type":"ContainerDied","Data":"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"}
Oct 30 12:43:30.634031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.633969    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff" event={"ID":"69f08044-0f0d-47ce-8640-b4593e1ee5c1","Type":"ContainerDied","Data":"65658e3359cbdfd865aeff384a5a9e1806e01ce89032f2f881f87cfd175dfb1b"}
Oct 30 12:43:30.634031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.633982    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"
Oct 30 12:43:30.634031 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.634001    4453 scope.go:122] "RemoveContainer" containerID="f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"
Oct 30 12:43:30.640001 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.639975    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56wx\" (UniqueName: \"kubernetes.io/projected/69f08044-0f0d-47ce-8640-b4593e1ee5c1-kube-api-access-x56wx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:30.640001 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.640001    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f08044-0f0d-47ce-8640-b4593e1ee5c1-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:30.640186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.640011    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:30.640186 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.640021    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69f08044-0f0d-47ce-8640-b4593e1ee5c1-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:30.644871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.644762    4453 scope.go:122] "RemoveContainer" containerID="f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"
Oct 30 12:43:30.645140 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:30.645115    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9\": container with ID starting with f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9 not found: ID does not exist" containerID="f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"
Oct 30 12:43:30.645237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.645147    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9"} err="failed to get container status \"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9\": rpc error: code = NotFound desc = could not find container \"f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9\": container with ID starting with f076d90092de8c37e586c2ebac98a47ae7ba778912fa4ea5395b33203be62de9 not found: ID does not exist"
Oct 30 12:43:30.680427 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.680348    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 12:43:30.684896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.684870    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-849669-75c786945d-k9xff"]
Oct 30 12:43:30.933497 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:30.933409    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" path="/var/lib/kubelet/pods/69f08044-0f0d-47ce-8640-b4593e1ee5c1/volumes"
Oct 30 12:43:35.588672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:35.587762    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:43:35.588672 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:35.588190    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy" containerID="cri-o://4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c" gracePeriod=30
Oct 30 12:43:37.571009 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:37.570969    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:43:40.862536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.862506    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:43:40.963601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963515    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.963601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963567    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.963816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963670    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.963816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963758    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.963816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963787    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.963977 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963819    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.964048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963904    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:43:40.964048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963938    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.964048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.963994    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:43:40.964231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964153    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.964231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964211    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\") pod \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\" (UID: \"8570a056-36cc-4e15-81b1-8d5527f3a7a2\") "
Oct 30 12:43:40.964398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964355    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:43:40.964640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964615    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:43:40.964799 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964778    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:43:40.965005 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.964989    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.965143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.965010    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.965143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.965025    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.965143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.965038    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.965143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.965052    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:40.966180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.966153    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:43:40.966307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.966181    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "kube-api-access-9xjjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:40.966307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.966251    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:43:40.966768 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:40.966744    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token" pod "8570a056-36cc-4e15-81b1-8d5527f3a7a2" (UID: "8570a056-36cc-4e15-81b1-8d5527f3a7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:43:41.066883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.066854    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:41.066883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.066880    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:41.066883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.066889    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:41.067121 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.066898    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjjx\" (UniqueName: \"kubernetes.io/projected/8570a056-36cc-4e15-81b1-8d5527f3a7a2-kube-api-access-9xjjx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:43:41.700786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.700751    4453 generic.go:356] "Generic (PLEG): container finished" podID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerID="4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c" exitCode=0
Oct 30 12:43:41.700961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.700802    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" event={"ID":"8570a056-36cc-4e15-81b1-8d5527f3a7a2","Type":"ContainerDied","Data":"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"}
Oct 30 12:43:41.700961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.700813    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"
Oct 30 12:43:41.700961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.700883    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2" event={"ID":"8570a056-36cc-4e15-81b1-8d5527f3a7a2","Type":"ContainerDied","Data":"e8b225a59dd49566e79bbc95d11783c497bd5721912d28e0cd768498cf1f1c2a"}
Oct 30 12:43:41.700961 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.700907    4453 scope.go:122] "RemoveContainer" containerID="4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"
Oct 30 12:43:41.711372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.711354    4453 scope.go:122] "RemoveContainer" containerID="4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"
Oct 30 12:43:41.711667 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:43:41.711645    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c\": container with ID starting with 4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c not found: ID does not exist" containerID="4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"
Oct 30 12:43:41.711716 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.711675    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c"} err="failed to get container status \"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c\": rpc error: code = NotFound desc = could not find container \"4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c\": container with ID starting with 4a6d9ad31ec93e88230feac3247a7f1e9ecc59d45c0331db79616f1733f7e62c not found: ID does not exist"
Oct 30 12:43:41.754880 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.754851    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:43:41.763995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:41.763969    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tjjd2"]
Oct 30 12:43:42.911763 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:42.911718    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" path="/var/lib/kubelet/pods/8570a056-36cc-4e15-81b1-8d5527f3a7a2/volumes"
Oct 30 12:43:59.044947 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:43:59.044918    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:44:00.797933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:00.797900    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:44:00.798376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:00.798101    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api" containerID="cri-o://c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe" gracePeriod=30
Oct 30 12:44:01.135534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.135509    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:44:01.154651 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.154621    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\" (UniqueName: \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\") pod \"71bbce70-cf88-4c88-8952-ec2f3797e382\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") "
Oct 30 12:44:01.154805 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.154722    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\") pod \"71bbce70-cf88-4c88-8952-ec2f3797e382\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") "
Oct 30 12:44:01.154862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.154847    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\") pod \"71bbce70-cf88-4c88-8952-ec2f3797e382\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") "
Oct 30 12:44:01.154916 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.154901    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\") pod \"71bbce70-cf88-4c88-8952-ec2f3797e382\" (UID: \"71bbce70-cf88-4c88-8952-ec2f3797e382\") "
Oct 30 12:44:01.155242 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.155217    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca" pod "71bbce70-cf88-4c88-8952-ec2f3797e382" (UID: "71bbce70-cf88-4c88-8952-ec2f3797e382"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:01.155367 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.155355    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71bbce70-cf88-4c88-8952-ec2f3797e382-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:01.156896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.156871    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs" pod "71bbce70-cf88-4c88-8952-ec2f3797e382" (UID: "71bbce70-cf88-4c88-8952-ec2f3797e382"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:01.157211 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.157186    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls" pod "71bbce70-cf88-4c88-8952-ec2f3797e382" (UID: "71bbce70-cf88-4c88-8952-ec2f3797e382"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:01.157307 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.157236    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v" pod "71bbce70-cf88-4c88-8952-ec2f3797e382" (UID: "71bbce70-cf88-4c88-8952-ec2f3797e382"). InnerVolumeSpecName "kube-api-access-nng6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:01.265462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.265416    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:01.265462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.265451    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nng6v\" (UniqueName: \"kubernetes.io/projected/71bbce70-cf88-4c88-8952-ec2f3797e382-kube-api-access-nng6v\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:01.265462 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.265465    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71bbce70-cf88-4c88-8952-ec2f3797e382-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:01.828685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.828646    4453 generic.go:356] "Generic (PLEG): container finished" podID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerID="c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe" exitCode=0
Oct 30 12:44:01.829167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.828717    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"
Oct 30 12:44:01.829167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.828730    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerDied","Data":"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"}
Oct 30 12:44:01.829167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.828838    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc" event={"ID":"71bbce70-cf88-4c88-8952-ec2f3797e382","Type":"ContainerDied","Data":"c38c17c78884dbfec0c6268c3eae4feafc05151f40654945819a69a9f571bbff"}
Oct 30 12:44:01.829167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.828872    4453 scope.go:122] "RemoveContainer" containerID="c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"
Oct 30 12:44:01.840970 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.840952    4453 scope.go:122] "RemoveContainer" containerID="834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"
Oct 30 12:44:01.849558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.849540    4453 scope.go:122] "RemoveContainer" containerID="c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"
Oct 30 12:44:01.849857 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:01.849837    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe\": container with ID starting with c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe not found: ID does not exist" containerID="c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"
Oct 30 12:44:01.849943 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.849866    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe"} err="failed to get container status \"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe\": rpc error: code = NotFound desc = could not find container \"c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe\": container with ID starting with c6765579da32948699395eb01b8ad935b91b2ee74a4213f50feb5b7caebe66fe not found: ID does not exist"
Oct 30 12:44:01.849943 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.849893    4453 scope.go:122] "RemoveContainer" containerID="834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"
Oct 30 12:44:01.850146 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:01.850124    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52\": container with ID starting with 834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52 not found: ID does not exist" containerID="834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"
Oct 30 12:44:01.850193 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.850153    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52"} err="failed to get container status \"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52\": rpc error: code = NotFound desc = could not find container \"834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52\": container with ID starting with 834db493c9eaac6f258f108d8d40e0f256e4fe0f3fee871a02389d2efc75eb52 not found: ID does not exist"
Oct 30 12:44:01.877447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.877421    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:44:01.882542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:01.882524    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-67e819-57777f7b47-vxtxc"]
Oct 30 12:44:02.910816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:02.910785    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" path="/var/lib/kubelet/pods/71bbce70-cf88-4c88-8952-ec2f3797e382/volumes"
Oct 30 12:44:04.046109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:04.046068    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing" containerID="cri-o://d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5" gracePeriod=25
Oct 30 12:44:07.570155 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:07.570114    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:44:10.187691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:10.187652    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:18.738370 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:18.738266    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:44:20.187955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:20.187913    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:20.945761 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:20.945726    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:44:20.980782 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:20.980745    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:44:24.320416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.320390    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:44:24.455909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.455875    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\" (UniqueName: \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\") pod \"9148c9db-ae77-4091-a0ef-31af5a722d08\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") "
Oct 30 12:44:24.456103 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.456083    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\") pod \"9148c9db-ae77-4091-a0ef-31af5a722d08\" (UID: \"9148c9db-ae77-4091-a0ef-31af5a722d08\") "
Oct 30 12:44:24.456505 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.456475    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume" pod "9148c9db-ae77-4091-a0ef-31af5a722d08" (UID: "9148c9db-ae77-4091-a0ef-31af5a722d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:24.456630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.456576    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9148c9db-ae77-4091-a0ef-31af5a722d08-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:24.458014 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.457994    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7" pod "9148c9db-ae77-4091-a0ef-31af5a722d08" (UID: "9148c9db-ae77-4091-a0ef-31af5a722d08"). InnerVolumeSpecName "kube-api-access-244d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:24.558275 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.558194    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-244d7\" (UniqueName: \"kubernetes.io/projected/9148c9db-ae77-4091-a0ef-31af5a722d08-kube-api-access-244d7\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:24.977037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.977000    4453 generic.go:356] "Generic (PLEG): container finished" podID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerID="d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5" exitCode=0
Oct 30 12:44:24.977215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.977049    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"
Oct 30 12:44:24.977215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.977080    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" event={"ID":"9148c9db-ae77-4091-a0ef-31af5a722d08","Type":"ContainerDied","Data":"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"}
Oct 30 12:44:24.977215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.977139    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb" event={"ID":"9148c9db-ae77-4091-a0ef-31af5a722d08","Type":"ContainerDied","Data":"11573a15c33b72a40b07c0c25c86a022f1e5586cea3dd780f9c3ad542111b46c"}
Oct 30 12:44:24.977215 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.977162    4453 scope.go:122] "RemoveContainer" containerID="d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"
Oct 30 12:44:24.986664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.986645    4453 scope.go:122] "RemoveContainer" containerID="d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"
Oct 30 12:44:24.986944 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:24.986921    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5\": container with ID starting with d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5 not found: ID does not exist" containerID="d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"
Oct 30 12:44:24.986995 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:24.986953    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5"} err="failed to get container status \"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5\": rpc error: code = NotFound desc = could not find container \"d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5\": container with ID starting with d1fb40291894cd03e4423ec671c1df942496cc86851f2ec3ebc2a811b22a7aa5 not found: ID does not exist"
Oct 30 12:44:25.021673 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:25.021645    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:44:25.030318 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:25.030295    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-67e819-5f49985cdc-rqhrb"]
Oct 30 12:44:25.946553 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:25.946512    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" podUID="23c6cc14-c200-4493-97a2-92b5d7f70139" containerName="payload-processing" containerID="cri-o://de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e" gracePeriod=25
Oct 30 12:44:25.981248 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:25.981207    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" podUID="def703c0-80d5-43e8-b458-2e71c4532e84" containerName="payload-pre-processing" containerID="cri-o://afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005" gracePeriod=25
Oct 30 12:44:26.255690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.255661    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:44:26.280180 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.280155    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:44:26.379106 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379070    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\" (UniqueName: \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\") pod \"23c6cc14-c200-4493-97a2-92b5d7f70139\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") "
Oct 30 12:44:26.379298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379176    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\") pod \"def703c0-80d5-43e8-b458-2e71c4532e84\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") "
Oct 30 12:44:26.379298 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379250    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\") pod \"23c6cc14-c200-4493-97a2-92b5d7f70139\" (UID: \"23c6cc14-c200-4493-97a2-92b5d7f70139\") "
Oct 30 12:44:26.379519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379400    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\" (UniqueName: \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\") pod \"def703c0-80d5-43e8-b458-2e71c4532e84\" (UID: \"def703c0-80d5-43e8-b458-2e71c4532e84\") "
Oct 30 12:44:26.379587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379570    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume" pod "def703c0-80d5-43e8-b458-2e71c4532e84" (UID: "def703c0-80d5-43e8-b458-2e71c4532e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:26.379677 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379655    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume" pod "23c6cc14-c200-4493-97a2-92b5d7f70139" (UID: "23c6cc14-c200-4493-97a2-92b5d7f70139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:26.379891 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379876    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def703c0-80d5-43e8-b458-2e71c4532e84-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:26.379958 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.379896    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6cc14-c200-4493-97a2-92b5d7f70139-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:26.381340 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.381308    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928" pod "23c6cc14-c200-4493-97a2-92b5d7f70139" (UID: "23c6cc14-c200-4493-97a2-92b5d7f70139"). InnerVolumeSpecName "kube-api-access-hh928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:26.381414 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.381373    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h" pod "def703c0-80d5-43e8-b458-2e71c4532e84" (UID: "def703c0-80d5-43e8-b458-2e71c4532e84"). InnerVolumeSpecName "kube-api-access-srh7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:26.481105 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.481033    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh7h\" (UniqueName: \"kubernetes.io/projected/def703c0-80d5-43e8-b458-2e71c4532e84-kube-api-access-srh7h\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:26.481105 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.481063    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh928\" (UniqueName: \"kubernetes.io/projected/23c6cc14-c200-4493-97a2-92b5d7f70139-kube-api-access-hh928\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:26.912517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.912483    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" path="/var/lib/kubelet/pods/9148c9db-ae77-4091-a0ef-31af5a722d08/volumes"
Oct 30 12:44:26.989845 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.989811    4453 generic.go:356] "Generic (PLEG): container finished" podID="23c6cc14-c200-4493-97a2-92b5d7f70139" containerID="de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e" exitCode=0
Oct 30 12:44:26.990239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.989856    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"
Oct 30 12:44:26.990239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.989915    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" event={"ID":"23c6cc14-c200-4493-97a2-92b5d7f70139","Type":"ContainerDied","Data":"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"}
Oct 30 12:44:26.990239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.989966    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w" event={"ID":"23c6cc14-c200-4493-97a2-92b5d7f70139","Type":"ContainerDied","Data":"e00f572374498eabd8b82a4186ab0225c1df17723bc036febe181f470877c923"}
Oct 30 12:44:26.990239 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.989996    4453 scope.go:122] "RemoveContainer" containerID="de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"
Oct 30 12:44:26.991407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.991385    4453 generic.go:356] "Generic (PLEG): container finished" podID="def703c0-80d5-43e8-b458-2e71c4532e84" containerID="afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005" exitCode=0
Oct 30 12:44:26.991517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.991463    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" event={"ID":"def703c0-80d5-43e8-b458-2e71c4532e84","Type":"ContainerDied","Data":"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"}
Oct 30 12:44:26.991517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.991499    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs" event={"ID":"def703c0-80d5-43e8-b458-2e71c4532e84","Type":"ContainerDied","Data":"a2199a89f8aecfd58de93b51d0279cb1676326a5fc2a13f4c619fbb3b077b0e8"}
Oct 30 12:44:26.991634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.991560    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"
Oct 30 12:44:26.999634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.999618    4453 scope.go:122] "RemoveContainer" containerID="de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"
Oct 30 12:44:26.999899 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:26.999880    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e\": container with ID starting with de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e not found: ID does not exist" containerID="de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"
Oct 30 12:44:26.999953 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.999935    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e"} err="failed to get container status \"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e\": rpc error: code = NotFound desc = could not find container \"de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e\": container with ID starting with de210976fa04065c45ff3861840bde3b569595e6b7076b9b6727d284a423df5e not found: ID does not exist"
Oct 30 12:44:27.000002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:26.999955    4453 scope.go:122] "RemoveContainer" containerID="afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"
Oct 30 12:44:27.008486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.008457    4453 scope.go:122] "RemoveContainer" containerID="afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"
Oct 30 12:44:27.008762 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:27.008734    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005\": container with ID starting with afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005 not found: ID does not exist" containerID="afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"
Oct 30 12:44:27.008865 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.008770    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005"} err="failed to get container status \"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005\": rpc error: code = NotFound desc = could not find container \"afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005\": container with ID starting with afecf4bb52c38c5ee7c4dd6fcb9f12c272dd00bc0e9bda986f230d95bc5ee005 not found: ID does not exist"
Oct 30 12:44:27.036474 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.036446    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:44:27.046543 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.046513    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bae155-56867f4547-67cxs"]
Oct 30 12:44:27.080766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.080721    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:44:27.085372 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:27.085309    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bae155-57bd48f8d-hrc7w"]
Oct 30 12:44:28.912085 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:28.912049    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c6cc14-c200-4493-97a2-92b5d7f70139" path="/var/lib/kubelet/pods/23c6cc14-c200-4493-97a2-92b5d7f70139/volumes"
Oct 30 12:44:28.912558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:28.912540    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def703c0-80d5-43e8-b458-2e71c4532e84" path="/var/lib/kubelet/pods/def703c0-80d5-43e8-b458-2e71c4532e84/volumes"
Oct 30 12:44:32.884230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:32.884193    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:44:32.884762 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:32.884470    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" podUID="98444241-0a51-467c-9446-16485e10d06f" containerName="maas-api" containerID="cri-o://665a7b1bebab8156e252dc922c793174556063ebfa6518626092670f083690e7" gracePeriod=30
Oct 30 12:44:33.031527 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.031487    4453 generic.go:356] "Generic (PLEG): container finished" podID="98444241-0a51-467c-9446-16485e10d06f" containerID="665a7b1bebab8156e252dc922c793174556063ebfa6518626092670f083690e7" exitCode=0
Oct 30 12:44:33.031694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.031528    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" event={"ID":"98444241-0a51-467c-9446-16485e10d06f","Type":"ContainerDied","Data":"665a7b1bebab8156e252dc922c793174556063ebfa6518626092670f083690e7"}
Oct 30 12:44:33.184900 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.184867    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:44:33.278660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.278626    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\") pod \"98444241-0a51-467c-9446-16485e10d06f\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") "
Oct 30 12:44:33.278846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.278692    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\") pod \"98444241-0a51-467c-9446-16485e10d06f\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") "
Oct 30 12:44:33.278846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.278760    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\" (UniqueName: \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\") pod \"98444241-0a51-467c-9446-16485e10d06f\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") "
Oct 30 12:44:33.278846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.278823    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\") pod \"98444241-0a51-467c-9446-16485e10d06f\" (UID: \"98444241-0a51-467c-9446-16485e10d06f\") "
Oct 30 12:44:33.279282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.279249    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca" pod "98444241-0a51-467c-9446-16485e10d06f" (UID: "98444241-0a51-467c-9446-16485e10d06f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:33.280955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.280928    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls" pod "98444241-0a51-467c-9446-16485e10d06f" (UID: "98444241-0a51-467c-9446-16485e10d06f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:33.281079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.280948    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs" pod "98444241-0a51-467c-9446-16485e10d06f" (UID: "98444241-0a51-467c-9446-16485e10d06f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:33.281079 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.280955    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt" pod "98444241-0a51-467c-9446-16485e10d06f" (UID: "98444241-0a51-467c-9446-16485e10d06f"). InnerVolumeSpecName "kube-api-access-hkctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:33.380521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.380489    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:33.380521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.380515    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98444241-0a51-467c-9446-16485e10d06f-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:33.380521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.380525    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkctt\" (UniqueName: \"kubernetes.io/projected/98444241-0a51-467c-9446-16485e10d06f-kube-api-access-hkctt\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:33.380764 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.380535    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98444241-0a51-467c-9446-16485e10d06f-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:33.760532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.760465    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerName="main" containerID="cri-o://c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272" gracePeriod=45
Oct 30 12:44:33.820774 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:33.820729    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": dial tcp 10.133.0.194:8000: connect: connection refused"
Oct 30 12:44:34.029574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.029517    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:44:34.037743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.037722    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"
Oct 30 12:44:34.037899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.037759    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" event={"ID":"98444241-0a51-467c-9446-16485e10d06f","Type":"ContainerDied","Data":"67136245ab981bb6a6af65a506c87b51b72de633dae1963324e5f76565b606af"}
Oct 30 12:44:34.037899 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.037817    4453 scope.go:122] "RemoveContainer" containerID="665a7b1bebab8156e252dc922c793174556063ebfa6518626092670f083690e7"
Oct 30 12:44:34.039398 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.039371    4453 generic.go:356] "Generic (PLEG): container finished" podID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerID="c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272" exitCode=0
Oct 30 12:44:34.039519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.039451    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" event={"ID":"9f7b7ce4-d5ed-4065-a6af-a5de8502d541","Type":"ContainerDied","Data":"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"}
Oct 30 12:44:34.039519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.039469    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"
Oct 30 12:44:34.039519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.039480    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf" event={"ID":"9f7b7ce4-d5ed-4065-a6af-a5de8502d541","Type":"ContainerDied","Data":"c8300a17c4961824d2f85250cd673d00d2de59d96c28cee9ee4abc77fcaca203"}
Oct 30 12:44:34.047417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.047401    4453 scope.go:122] "RemoveContainer" containerID="c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"
Oct 30 12:44:34.056877 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.056862    4453 scope.go:122] "RemoveContainer" containerID="c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"
Oct 30 12:44:34.057178 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:34.057160    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272\": container with ID starting with c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272 not found: ID does not exist" containerID="c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"
Oct 30 12:44:34.057234 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.057186    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272"} err="failed to get container status \"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272\": rpc error: code = NotFound desc = could not find container \"c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272\": container with ID starting with c9b71a0af9a8b47c0fda0a75eb26577501ad479c66556451a5f52e6a3a9ef272 not found: ID does not exist"
Oct 30 12:44:34.096383 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.096320    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:44:34.102602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.102576    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh"]
Oct 30 12:44:34.188714 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.188679    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.188926 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.188839    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.188926 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.188911    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.189041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.188940    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\" (UniqueName: \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.189041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.188996    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:34.189041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189018    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.189198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189075    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\") pod \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\" (UID: \"9f7b7ce4-d5ed-4065-a6af-a5de8502d541\") "
Oct 30 12:44:34.189198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189100    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:34.189198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189137    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:34.189560 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189542    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.189629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189561    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.189629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.189571    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.191377 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.191321    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:34.191607 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.191588    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:44:34.191661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.191611    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf" pod "9f7b7ce4-d5ed-4065-a6af-a5de8502d541" (UID: "9f7b7ce4-d5ed-4065-a6af-a5de8502d541"). InnerVolumeSpecName "kube-api-access-zg7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:34.290306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.290227    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg7bf\" (UniqueName: \"kubernetes.io/projected/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-kube-api-access-zg7bf\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.290306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.290257    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.290306 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.290268    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7b7ce4-d5ed-4065-a6af-a5de8502d541-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:34.380187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.380151    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:44:34.382200 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.382176    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-2c719a/rate-test-model-2c719a-kserve-657bd586df-hp7kf"]
Oct 30 12:44:34.912262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.912228    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98444241-0a51-467c-9446-16485e10d06f" path="/var/lib/kubelet/pods/98444241-0a51-467c-9446-16485e10d06f/volumes"
Oct 30 12:44:34.912675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:34.912660    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" path="/var/lib/kubelet/pods/9f7b7ce4-d5ed-4065-a6af-a5de8502d541/volumes"
Oct 30 12:44:36.179360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:36.177420    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bae155-57dd77cccc-h2bdh" podUID="98444241-0a51-467c-9446-16485e10d06f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": context deadline exceeded"
Oct 30 12:44:37.569679 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:37.569642    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:44:38.427457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:38.427421    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:44:38.451444 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:38.451405    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:44:43.428074 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:43.428001    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing" containerID="cri-o://23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a" gracePeriod=25
Oct 30 12:44:43.452584 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:43.452532    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing" containerID="cri-o://3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9" gracePeriod=25
Oct 30 12:44:45.105782 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.105747    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:44:45.106693 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.106634    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" podUID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" containerName="maas-api" containerID="cri-o://08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3" gracePeriod=30
Oct 30 12:44:45.399378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.396617    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:44:45.413620 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.413582    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\") pod \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") "
Oct 30 12:44:45.413804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.413652    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\") pod \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") "
Oct 30 12:44:45.413804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.413788    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\" (UniqueName: \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\") pod \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") "
Oct 30 12:44:45.413931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.413814    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\") pod \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\" (UID: \"cf758a8d-c9e6-42f0-a655-99fe55aaa01e\") "
Oct 30 12:44:45.414103 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.414074    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca" pod "cf758a8d-c9e6-42f0-a655-99fe55aaa01e" (UID: "cf758a8d-c9e6-42f0-a655-99fe55aaa01e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:44:45.414378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.414325    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.416204 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.416176    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls" pod "cf758a8d-c9e6-42f0-a655-99fe55aaa01e" (UID: "cf758a8d-c9e6-42f0-a655-99fe55aaa01e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:45.416296 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.416226    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t" pod "cf758a8d-c9e6-42f0-a655-99fe55aaa01e" (UID: "cf758a8d-c9e6-42f0-a655-99fe55aaa01e"). InnerVolumeSpecName "kube-api-access-xzn9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:44:45.416661 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.416638    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs" pod "cf758a8d-c9e6-42f0-a655-99fe55aaa01e" (UID: "cf758a8d-c9e6-42f0-a655-99fe55aaa01e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:44:45.515457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.515406    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.515457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.515452    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzn9t\" (UniqueName: \"kubernetes.io/projected/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-kube-api-access-xzn9t\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.515457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.515462    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf758a8d-c9e6-42f0-a655-99fe55aaa01e-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:44:45.887545 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.887506    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:45.887703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:45.887506    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:46.145893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.145789    4453 generic.go:356] "Generic (PLEG): container finished" podID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" containerID="08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3" exitCode=0
Oct 30 12:44:46.145893 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.145835    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"
Oct 30 12:44:46.146453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.145878    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" event={"ID":"cf758a8d-c9e6-42f0-a655-99fe55aaa01e","Type":"ContainerDied","Data":"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"}
Oct 30 12:44:46.146453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.145951    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj" event={"ID":"cf758a8d-c9e6-42f0-a655-99fe55aaa01e","Type":"ContainerDied","Data":"8b5a3637e1ef26d82f81343d2e7713da21aabdeff6f9e91c127ef97ee3f036f7"}
Oct 30 12:44:46.146453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.145964    4453 scope.go:122] "RemoveContainer" containerID="08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"
Oct 30 12:44:46.157290 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.157258    4453 scope.go:122] "RemoveContainer" containerID="08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"
Oct 30 12:44:46.157607 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:44:46.157581    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3\": container with ID starting with 08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3 not found: ID does not exist" containerID="08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"
Oct 30 12:44:46.157675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.157618    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3"} err="failed to get container status \"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3\": rpc error: code = NotFound desc = could not find container \"08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3\": container with ID starting with 08ee9a59ed587f6b62fb237a1ac7953babdd7d5309b67bc2e37c2dc5751841b3 not found: ID does not exist"
Oct 30 12:44:46.190871 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.190841    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:44:46.199915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.199890    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-846099-5d468b4c6d-8prwj"]
Oct 30 12:44:46.914206 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:46.914168    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" path="/var/lib/kubelet/pods/cf758a8d-c9e6-42f0-a655-99fe55aaa01e/volumes"
Oct 30 12:44:55.888235 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:55.888194    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:44:55.888839 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:44:55.888187    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:00.164954 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.164920    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:00.165492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165465    4453 cpu_manager.go:414] "containerMap: removing container" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.165492 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165486    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165501    4453 cpu_manager.go:414] "containerMap: removing container" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165507    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165523    4453 cpu_manager.go:414] "containerMap: removing container" podUID="def703c0-80d5-43e8-b458-2e71c4532e84" containerName="payload-pre-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165528    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def703c0-80d5-43e8-b458-2e71c4532e84" containerName="payload-pre-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165539    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165544    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165557    4453 cpu_manager.go:414] "containerMap: removing container" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165564    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165581    4453 cpu_manager.go:414] "containerMap: removing container" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerName="main"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165588    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerName="main"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165600    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" containerName="test-curl-7377-edefe4"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165607    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" containerName="test-curl-7377-edefe4"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165623    4453 cpu_manager.go:414] "containerMap: removing container" podUID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165631    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165640    4453 cpu_manager.go:414] "containerMap: removing container" podUID="23c6cc14-c200-4493-97a2-92b5d7f70139" containerName="payload-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165645    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c6cc14-c200-4493-97a2-92b5d7f70139" containerName="payload-processing"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165654    4453 cpu_manager.go:414] "containerMap: removing container" podUID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165659    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165669    4453 cpu_manager.go:414] "containerMap: removing container" podUID="98444241-0a51-467c-9446-16485e10d06f" containerName="maas-api"
Oct 30 12:45:00.165682 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165674    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98444241-0a51-467c-9446-16485e10d06f" containerName="maas-api"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165785    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f08044-0f0d-47ce-8640-b4593e1ee5c1" containerName="maas-api"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165794    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c6cc14-c200-4493-97a2-92b5d7f70139" containerName="payload-processing"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165801    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf758a8d-c9e6-42f0-a655-99fe55aaa01e" containerName="maas-api"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165808    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="98444241-0a51-467c-9446-16485e10d06f" containerName="maas-api"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165816    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165822    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="9148c9db-ae77-4091-a0ef-31af5a722d08" containerName="payload-pre-processing"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165830    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="def703c0-80d5-43e8-b458-2e71c4532e84" containerName="payload-pre-processing"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165836    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8570a056-36cc-4e15-81b1-8d5527f3a7a2" containerName="istio-proxy"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165844    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7b7ce4-d5ed-4065-a6af-a5de8502d541" containerName="main"
Oct 30 12:45:00.166371 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.165851    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19f7b61-1de2-42f5-9001-4e7e263fa520" containerName="test-curl-7377-edefe4"
Oct 30 12:45:00.174410 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.172558    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.211262 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.211227    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:00.218350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.213578    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:00.218350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.214237    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bbce70-cf88-4c88-8952-ec2f3797e382" containerName="maas-api"
Oct 30 12:45:00.218350 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.218323    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.233002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.232972    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:00.286857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.286822    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-k9f5r\""
Oct 30 12:45:00.293606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.293579    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcll2\" (UniqueName: \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.293707 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.293689    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.394875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.394836    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.395068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.394900    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshqt\" (UniqueName: \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.395068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.395008    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcll2\" (UniqueName: \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.395143 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.395090    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.395756 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.395738    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.416081 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.416018    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcll2\" (UniqueName: \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.495881 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.495834    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:00.496208 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.496179    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.496291 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.496275    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshqt\" (UniqueName: \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.496874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.496851    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.517786 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.517756    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshqt\" (UniqueName: \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.544521 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.544475    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:00.792685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.792658    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:00.795029 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:45:00.795007    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e680947_7fdd_4f28_a956_148150b3c1ac.slice/crio-badcf03abb9541e5a341562613f646b9391696f551a15f15ad457195bfca4eb7 WatchSource:0}: Error finding container badcf03abb9541e5a341562613f646b9391696f551a15f15ad457195bfca4eb7: Status 404 returned error can't find the container with id badcf03abb9541e5a341562613f646b9391696f551a15f15ad457195bfca4eb7
Oct 30 12:45:00.861109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:00.861086    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:00.863112 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:45:00.863085    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01281026_c67e_4b92_baa9_fe0215e1a027.slice/crio-6346ff078f93f0684eedb7c4afcf4c2bc3c5a24c5f771244906aaf49dbc3726c WatchSource:0}: Error finding container 6346ff078f93f0684eedb7c4afcf4c2bc3c5a24c5f771244906aaf49dbc3726c: Status 404 returned error can't find the container with id 6346ff078f93f0684eedb7c4afcf4c2bc3c5a24c5f771244906aaf49dbc3726c
Oct 30 12:45:01.229921 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.229877    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerStarted","Data":"b093a4660003d5bbc9b1cf0843ef7605517368ebe22f5d35a0e3d491d2d790b8"}
Oct 30 12:45:01.230411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.229942    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerStarted","Data":"6346ff078f93f0684eedb7c4afcf4c2bc3c5a24c5f771244906aaf49dbc3726c"}
Oct 30 12:45:01.231446 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.231417    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerStarted","Data":"3b93dcda1b0731cca317bdd564fb37accc9e6d35fbd01ef74baf78fb05751fe7"}
Oct 30 12:45:01.231574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.231461    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerStarted","Data":"badcf03abb9541e5a341562613f646b9391696f551a15f15ad457195bfca4eb7"}
Oct 30 12:45:01.266419 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.266359    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" podStartSLOduration=1.266343526 podStartE2EDuration="1.266343526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.264183535 +0000 UTC m=+4500.484468444" watchObservedRunningTime="2026-10-30 12:45:01.266343526 +0000 UTC m=+4500.486628431"
Oct 30 12:45:01.302114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:01.302062    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" podStartSLOduration=1.302046236 podStartE2EDuration="1.302046236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:01.300093172 +0000 UTC m=+4500.520378083" watchObservedRunningTime="2026-10-30 12:45:01.302046236 +0000 UTC m=+4500.522331149"
Oct 30 12:45:02.248104 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.248064    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerID="3b93dcda1b0731cca317bdd564fb37accc9e6d35fbd01ef74baf78fb05751fe7" exitCode=60
Oct 30 12:45:02.248641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.248135    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerDied","Data":"3b93dcda1b0731cca317bdd564fb37accc9e6d35fbd01ef74baf78fb05751fe7"}
Oct 30 12:45:02.248641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.248536    4453 scope.go:122] "RemoveContainer" containerID="3b93dcda1b0731cca317bdd564fb37accc9e6d35fbd01ef74baf78fb05751fe7"
Oct 30 12:45:02.250705 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.250673    4453 generic.go:356] "Generic (PLEG): container finished" podID="01281026-c67e-4b92-baa9-fe0215e1a027" containerID="b093a4660003d5bbc9b1cf0843ef7605517368ebe22f5d35a0e3d491d2d790b8" exitCode=60
Oct 30 12:45:02.250810 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.250723    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerDied","Data":"b093a4660003d5bbc9b1cf0843ef7605517368ebe22f5d35a0e3d491d2d790b8"}
Oct 30 12:45:02.250988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:02.250973    4453 scope.go:122] "RemoveContainer" containerID="b093a4660003d5bbc9b1cf0843ef7605517368ebe22f5d35a0e3d491d2d790b8"
Oct 30 12:45:03.257984 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.257951    4453 generic.go:356] "Generic (PLEG): container finished" podID="01281026-c67e-4b92-baa9-fe0215e1a027" containerID="68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c" exitCode=60
Oct 30 12:45:03.258502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.258035    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerDied","Data":"68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c"}
Oct 30 12:45:03.258502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.258101    4453 scope.go:122] "RemoveContainer" containerID="b093a4660003d5bbc9b1cf0843ef7605517368ebe22f5d35a0e3d491d2d790b8"
Oct 30 12:45:03.258502 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.258404    4453 scope.go:122] "RemoveContainer" containerID="68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c"
Oct 30 12:45:03.258746 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:03.258723    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f_odh-ai-gateway-infra(01281026-c67e-4b92-baa9-fe0215e1a027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" podUID="01281026-c67e-4b92-baa9-fe0215e1a027"
Oct 30 12:45:03.260049 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.260022    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerID="69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694" exitCode=60
Oct 30 12:45:03.260148 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.260068    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerDied","Data":"69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694"}
Oct 30 12:45:03.260326 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.260313    4453 scope.go:122] "RemoveContainer" containerID="69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694"
Oct 30 12:45:03.260584 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:03.260566    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j_odh-ai-gateway-infra(8e680947-7fdd-4f28-a956-148150b3c1ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac"
Oct 30 12:45:03.269806 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.269784    4453 scope.go:122] "RemoveContainer" containerID="3b93dcda1b0731cca317bdd564fb37accc9e6d35fbd01ef74baf78fb05751fe7"
Oct 30 12:45:03.716602 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.716574    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:45:03.748732 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.748710    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:45:03.830295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.830266    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\") pod \"91372140-3b06-4754-b790-b413ead12b0c\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") "
Oct 30 12:45:03.830493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.830323    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\" (UniqueName: \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\") pod \"91372140-3b06-4754-b790-b413ead12b0c\" (UID: \"91372140-3b06-4754-b790-b413ead12b0c\") "
Oct 30 12:45:03.830726 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.830705    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume" pod "91372140-3b06-4754-b790-b413ead12b0c" (UID: "91372140-3b06-4754-b790-b413ead12b0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:03.832448 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.832424    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx" pod "91372140-3b06-4754-b790-b413ead12b0c" (UID: "91372140-3b06-4754-b790-b413ead12b0c"). InnerVolumeSpecName "kube-api-access-fsptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:03.931631 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.931596    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\") pod \"ada18417-a16b-4041-8a0c-cf2746c19f6a\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") "
Oct 30 12:45:03.931805 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.931709    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\" (UniqueName: \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\") pod \"ada18417-a16b-4041-8a0c-cf2746c19f6a\" (UID: \"ada18417-a16b-4041-8a0c-cf2746c19f6a\") "
Oct 30 12:45:03.932068 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.932040    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91372140-3b06-4754-b790-b413ead12b0c-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:03.932182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.932074    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsptx\" (UniqueName: \"kubernetes.io/projected/91372140-3b06-4754-b790-b413ead12b0c-kube-api-access-fsptx\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:03.932182 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.932043    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume" pod "ada18417-a16b-4041-8a0c-cf2746c19f6a" (UID: "ada18417-a16b-4041-8a0c-cf2746c19f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:03.933826 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:03.933808    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q" pod "ada18417-a16b-4041-8a0c-cf2746c19f6a" (UID: "ada18417-a16b-4041-8a0c-cf2746c19f6a"). InnerVolumeSpecName "kube-api-access-2774q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:04.033089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.032990    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2774q\" (UniqueName: \"kubernetes.io/projected/ada18417-a16b-4041-8a0c-cf2746c19f6a-kube-api-access-2774q\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:04.033089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.033035    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ada18417-a16b-4041-8a0c-cf2746c19f6a-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:04.268403 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.268371    4453 scope.go:122] "RemoveContainer" containerID="68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c"
Oct 30 12:45:04.268887 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:04.268665    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f_odh-ai-gateway-infra(01281026-c67e-4b92-baa9-fe0215e1a027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" podUID="01281026-c67e-4b92-baa9-fe0215e1a027"
Oct 30 12:45:04.269556 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.269530    4453 generic.go:356] "Generic (PLEG): container finished" podID="91372140-3b06-4754-b790-b413ead12b0c" containerID="23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a" exitCode=0
Oct 30 12:45:04.269699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.269559    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" event={"ID":"91372140-3b06-4754-b790-b413ead12b0c","Type":"ContainerDied","Data":"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"}
Oct 30 12:45:04.269699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.269578    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"
Oct 30 12:45:04.269699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.269615    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv" event={"ID":"91372140-3b06-4754-b790-b413ead12b0c","Type":"ContainerDied","Data":"f6ec413aaaf15c8f6738d846223370655a2236a5797649f7e18bfca93105b35d"}
Oct 30 12:45:04.269699 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.269642    4453 scope.go:122] "RemoveContainer" containerID="23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"
Oct 30 12:45:04.272352 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.272321    4453 scope.go:122] "RemoveContainer" containerID="69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694"
Oct 30 12:45:04.272641 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:04.272618    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j_odh-ai-gateway-infra(8e680947-7fdd-4f28-a956-148150b3c1ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac"
Oct 30 12:45:04.273540 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.273516    4453 generic.go:356] "Generic (PLEG): container finished" podID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerID="3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9" exitCode=0
Oct 30 12:45:04.273625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.273555    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"
Oct 30 12:45:04.273625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.273574    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" event={"ID":"ada18417-a16b-4041-8a0c-cf2746c19f6a","Type":"ContainerDied","Data":"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"}
Oct 30 12:45:04.273625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.273612    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7" event={"ID":"ada18417-a16b-4041-8a0c-cf2746c19f6a","Type":"ContainerDied","Data":"6cd5de1ac2d861c40eb90397a5e93ce909d18656b3727e39b8a972c05130fab9"}
Oct 30 12:45:04.281597 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.281578    4453 scope.go:122] "RemoveContainer" containerID="23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"
Oct 30 12:45:04.281898 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:04.281871    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a\": container with ID starting with 23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a not found: ID does not exist" containerID="23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"
Oct 30 12:45:04.281979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.281909    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a"} err="failed to get container status \"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a\": rpc error: code = NotFound desc = could not find container \"23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a\": container with ID starting with 23ab5f4b8547683c42e66da137b018030b21ee0ab22a7ed3b733aa4c5bb8b19a not found: ID does not exist"
Oct 30 12:45:04.281979 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.281925    4453 scope.go:122] "RemoveContainer" containerID="3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"
Oct 30 12:45:04.291238 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.291220    4453 scope.go:122] "RemoveContainer" containerID="3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"
Oct 30 12:45:04.291662 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:04.291637    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9\": container with ID starting with 3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9 not found: ID does not exist" containerID="3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"
Oct 30 12:45:04.291767 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.291668    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9"} err="failed to get container status \"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9\": rpc error: code = NotFound desc = could not find container \"3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9\": container with ID starting with 3bd0e522e25598811f61ac26be0f491056361ffa1e974f9e32e95bc09e1ea7b9 not found: ID does not exist"
Oct 30 12:45:04.337500 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.337462    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:45:04.343089 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.343058    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846099-6cbbd9db47-wtrn7"]
Oct 30 12:45:04.369417 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.369389    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:45:04.377630 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.377598    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846099-56bff4c798-z7nkv"]
Oct 30 12:45:04.912537 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.912503    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91372140-3b06-4754-b790-b413ead12b0c" path="/var/lib/kubelet/pods/91372140-3b06-4754-b790-b413ead12b0c/volumes"
Oct 30 12:45:04.913111 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:04.913089    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" path="/var/lib/kubelet/pods/ada18417-a16b-4041-8a0c-cf2746c19f6a/volumes"
Oct 30 12:45:07.579902 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:07.579861    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:45:08.437525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.437485    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"]
Oct 30 12:45:08.438201 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438182    4453 cpu_manager.go:414] "containerMap: removing container" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing"
Oct 30 12:45:08.438259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438204    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing"
Oct 30 12:45:08.438259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438228    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing"
Oct 30 12:45:08.438259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438236    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing"
Oct 30 12:45:08.438423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438410    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="91372140-3b06-4754-b790-b413ead12b0c" containerName="payload-processing"
Oct 30 12:45:08.438459 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.438430    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada18417-a16b-4041-8a0c-cf2746c19f6a" containerName="payload-pre-processing"
Oct 30 12:45:08.443134 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.443113    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.460236 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.460213    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 12:45:08.480564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.480535    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lwg\" (UniqueName: \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.480694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.480597    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.480694 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.480670    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.494680 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.494653    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"]
Oct 30 12:45:08.581517 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.581483    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lwg\" (UniqueName: \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.581923 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.581552    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.581923 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.581578    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.582228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.582201    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.583931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.583913    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.607760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.607731    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lwg\" (UniqueName: \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:08.756564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:08.756490    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:09.055690 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:09.055662    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"]
Oct 30 12:45:09.057438 ip-10-0-143-18 kubenswrapper[4453]: W1030 12:45:09.057410    4453 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5afb0c_baac_474b_8c1e_a8aa8f52b67a.slice/crio-3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809 WatchSource:0}: Error finding container 3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809: Status 404 returned error can't find the container with id 3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809
Oct 30 12:45:09.304421 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:09.304359    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk" event={"ID":"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a","Type":"ContainerStarted","Data":"3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809"}
Oct 30 12:45:10.311850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:10.311764    4453 generic.go:356] "Generic (PLEG): container finished" podID="ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" containerID="6ea3ab9f17288913543c439a182ea9228b5cc0276ca1dd4ae882676e0456cdf9" exitCode=0
Oct 30 12:45:10.311850 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:10.311807    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk" event={"ID":"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a","Type":"ContainerDied","Data":"6ea3ab9f17288913543c439a182ea9228b5cc0276ca1dd4ae882676e0456cdf9"}
Oct 30 12:45:11.490310 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.490279    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:11.615441 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.615409    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\" (UniqueName: \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\") pod \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") "
Oct 30 12:45:11.615663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.615520    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\") pod \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") "
Oct 30 12:45:11.615663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.615569    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\") pod \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\" (UID: \"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a\") "
Oct 30 12:45:11.619913 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.619867    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg" pod "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" (UID: "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a"). InnerVolumeSpecName "kube-api-access-r9lwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:11.621187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.621154    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca" pod "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" (UID: "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:11.623933 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.623903    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca" pod "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" (UID: "ec5afb0c-baac-474b-8c1e-a8aa8f52b67a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:45:11.717691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.717650    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lwg\" (UniqueName: \"kubernetes.io/projected/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-kube-api-access-r9lwg\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:11.717691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.717686    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-xks-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:11.717874 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:11.717700    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a-maas-api-openshift-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:12.324610 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:12.324578    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"
Oct 30 12:45:12.324802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:12.324583    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk" event={"ID":"ec5afb0c-baac-474b-8c1e-a8aa8f52b67a","Type":"ContainerDied","Data":"3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809"}
Oct 30 12:45:12.324802 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:12.324722    4453 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3beb818bed55ce562b654dd235ef6b7d5b4366a29a42371da062e712ec041809"
Oct 30 12:45:15.905007 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:15.904975    4453 scope.go:122] "RemoveContainer" containerID="69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694"
Oct 30 12:45:15.905564 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:15.905076    4453 scope.go:122] "RemoveContainer" containerID="68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c"
Oct 30 12:45:16.346811 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:16.346776    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerStarted","Data":"70c2472a49ac31783c5224897e61d458bf5718c7911651f62ec6dda004496167"}
Oct 30 12:45:16.348650 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:16.348619    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerStarted","Data":"941ebbeb457961b7b13707e82197fee95d699bb7bbc84613cd202155715c0441"}
Oct 30 12:45:17.355429 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.355393    4453 generic.go:356] "Generic (PLEG): container finished" podID="01281026-c67e-4b92-baa9-fe0215e1a027" containerID="70c2472a49ac31783c5224897e61d458bf5718c7911651f62ec6dda004496167" exitCode=60
Oct 30 12:45:17.355928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.355472    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerDied","Data":"70c2472a49ac31783c5224897e61d458bf5718c7911651f62ec6dda004496167"}
Oct 30 12:45:17.355928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.355541    4453 scope.go:122] "RemoveContainer" containerID="68ba13ed7e2a53a126c091f47806d15dc8e7a72dcb41b4ec99f7aad300ccd85c"
Oct 30 12:45:17.355928 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.355921    4453 scope.go:122] "RemoveContainer" containerID="70c2472a49ac31783c5224897e61d458bf5718c7911651f62ec6dda004496167"
Oct 30 12:45:17.356260 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:17.356235    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f_odh-ai-gateway-infra(01281026-c67e-4b92-baa9-fe0215e1a027)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" podUID="01281026-c67e-4b92-baa9-fe0215e1a027"
Oct 30 12:45:17.358002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.357976    4453 generic.go:356] "Generic (PLEG): container finished" podID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerID="941ebbeb457961b7b13707e82197fee95d699bb7bbc84613cd202155715c0441" exitCode=60
Oct 30 12:45:17.358109 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.358014    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerDied","Data":"941ebbeb457961b7b13707e82197fee95d699bb7bbc84613cd202155715c0441"}
Oct 30 12:45:17.358399 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.358376    4453 scope.go:122] "RemoveContainer" containerID="941ebbeb457961b7b13707e82197fee95d699bb7bbc84613cd202155715c0441"
Oct 30 12:45:17.358696 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:17.358667    4453 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j_odh-ai-gateway-infra(8e680947-7fdd-4f28-a956-148150b3c1ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac"
Oct 30 12:45:17.385358 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.385300    4453 scope.go:122] "RemoveContainer" containerID="69ddaa6d38954e0857c4fa5a7f15ca6cde16b55d6c51c62e656f232126d0f694"
Oct 30 12:45:17.400942 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.400917    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:17.440059 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:17.440015    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:18.513061 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.513030    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:18.545449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.545423    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:18.599831 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.599794    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\" (UniqueName: \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\") pod \"8e680947-7fdd-4f28-a956-148150b3c1ac\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") "
Oct 30 12:45:18.600032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.599854    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\") pod \"01281026-c67e-4b92-baa9-fe0215e1a027\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") "
Oct 30 12:45:18.600032 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.599950    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\" (UniqueName: \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\") pod \"01281026-c67e-4b92-baa9-fe0215e1a027\" (UID: \"01281026-c67e-4b92-baa9-fe0215e1a027\") "
Oct 30 12:45:18.600177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.600051    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\") pod \"8e680947-7fdd-4f28-a956-148150b3c1ac\" (UID: \"8e680947-7fdd-4f28-a956-148150b3c1ac\") "
Oct 30 12:45:18.600717 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.600686    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca" pod "8e680947-7fdd-4f28-a956-148150b3c1ac" (UID: "8e680947-7fdd-4f28-a956-148150b3c1ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:18.600838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.600753    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca" pod "01281026-c67e-4b92-baa9-fe0215e1a027" (UID: "01281026-c67e-4b92-baa9-fe0215e1a027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:18.603514 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.603490    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2" pod "8e680947-7fdd-4f28-a956-148150b3c1ac" (UID: "8e680947-7fdd-4f28-a956-148150b3c1ac"). InnerVolumeSpecName "kube-api-access-rcll2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:18.603652 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.603583    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt" pod "01281026-c67e-4b92-baa9-fe0215e1a027" (UID: "01281026-c67e-4b92-baa9-fe0215e1a027"). InnerVolumeSpecName "kube-api-access-tshqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:18.701391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.701290    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e680947-7fdd-4f28-a956-148150b3c1ac-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:18.701391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.701319    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcll2\" (UniqueName: \"kubernetes.io/projected/8e680947-7fdd-4f28-a956-148150b3c1ac-kube-api-access-rcll2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:18.701391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.701355    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01281026-c67e-4b92-baa9-fe0215e1a027-maas-api-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:18.701391 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:18.701365    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tshqt\" (UniqueName: \"kubernetes.io/projected/01281026-c67e-4b92-baa9-fe0215e1a027-kube-api-access-tshqt\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:19.372409 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.372379    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"
Oct 30 12:45:19.372409 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.372402    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f" event={"ID":"01281026-c67e-4b92-baa9-fe0215e1a027","Type":"ContainerDied","Data":"6346ff078f93f0684eedb7c4afcf4c2bc3c5a24c5f771244906aaf49dbc3726c"}
Oct 30 12:45:19.372663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.372468    4453 scope.go:122] "RemoveContainer" containerID="70c2472a49ac31783c5224897e61d458bf5718c7911651f62ec6dda004496167"
Oct 30 12:45:19.373997 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.373972    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"
Oct 30 12:45:19.374116 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.373968    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j" event={"ID":"8e680947-7fdd-4f28-a956-148150b3c1ac","Type":"ContainerDied","Data":"badcf03abb9541e5a341562613f646b9391696f551a15f15ad457195bfca4eb7"}
Oct 30 12:45:19.383712 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.383690    4453 scope.go:122] "RemoveContainer" containerID="941ebbeb457961b7b13707e82197fee95d699bb7bbc84613cd202155715c0441"
Oct 30 12:45:19.424984 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.424960    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:19.434285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.434257    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4fff4f-29889405-qd85j"]
Oct 30 12:45:19.479760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.479732    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:19.497790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:19.497761    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3ccdab-29889405-s726f"]
Oct 30 12:45:20.910404 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:20.910370    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" path="/var/lib/kubelet/pods/01281026-c67e-4b92-baa9-fe0215e1a027/volumes"
Oct 30 12:45:20.910822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:20.910807    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" path="/var/lib/kubelet/pods/8e680947-7fdd-4f28-a956-148150b3c1ac/volumes"
Oct 30 12:45:28.663617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:28.663580    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:45:33.130036 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:33.130003    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:45:33.351950 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:33.351907    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"]
Oct 30 12:45:33.355168 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:33.355141    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f8f132-q9vwk"]
Oct 30 12:45:33.664139 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:33.664075    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing" containerID="cri-o://8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65" gracePeriod=25
Oct 30 12:45:34.910748 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:34.910710    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" path="/var/lib/kubelet/pods/ec5afb0c-baac-474b-8c1e-a8aa8f52b67a/volumes"
Oct 30 12:45:36.058890 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:36.058850    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:37.570495 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:37.570455    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:45:38.130749 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:38.130710    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing" containerID="cri-o://f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b" gracePeriod=25
Oct 30 12:45:38.621574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:38.621539    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:46.059246 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:46.059162    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:48.622289 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:48.622254    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:45:52.250565 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:52.250530    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:45:52.358593 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:52.358558    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 12:45:53.260571 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:53.260532    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 12:45:53.271076 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:53.271040    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 12:45:53.924760 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:53.924705    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:45:54.007241 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.007205    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\" (UniqueName: \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\") pod \"ec90dfa7-8dc3-4928-a13b-26571018407f\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") "
Oct 30 12:45:54.007463 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.007269    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\") pod \"ec90dfa7-8dc3-4928-a13b-26571018407f\" (UID: \"ec90dfa7-8dc3-4928-a13b-26571018407f\") "
Oct 30 12:45:54.007691 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.007667    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume" pod "ec90dfa7-8dc3-4928-a13b-26571018407f" (UID: "ec90dfa7-8dc3-4928-a13b-26571018407f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:54.007821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.007805    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec90dfa7-8dc3-4928-a13b-26571018407f-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:54.010064 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.010045    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8" pod "ec90dfa7-8dc3-4928-a13b-26571018407f" (UID: "ec90dfa7-8dc3-4928-a13b-26571018407f"). InnerVolumeSpecName "kube-api-access-z2jh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:54.109233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.109196    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jh8\" (UniqueName: \"kubernetes.io/projected/ec90dfa7-8dc3-4928-a13b-26571018407f-kube-api-access-z2jh8\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:54.567364 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.567298    4453 generic.go:356] "Generic (PLEG): container finished" podID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerID="8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65" exitCode=0
Oct 30 12:45:54.567896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.567381    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"
Oct 30 12:45:54.567896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.567493    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" event={"ID":"ec90dfa7-8dc3-4928-a13b-26571018407f","Type":"ContainerDied","Data":"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"}
Oct 30 12:45:54.567896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.567544    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72" event={"ID":"ec90dfa7-8dc3-4928-a13b-26571018407f","Type":"ContainerDied","Data":"fe6502aaa17e544ae94d1acf1d1541a890aa21feccb9cce602781dfd76795b97"}
Oct 30 12:45:54.567896 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.567573    4453 scope.go:122] "RemoveContainer" containerID="8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"
Oct 30 12:45:54.577495 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.577468    4453 scope.go:122] "RemoveContainer" containerID="8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"
Oct 30 12:45:54.577813 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:54.577793    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65\": container with ID starting with 8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65 not found: ID does not exist" containerID="8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"
Oct 30 12:45:54.577883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.577821    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65"} err="failed to get container status \"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65\": rpc error: code = NotFound desc = could not find container \"8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65\": container with ID starting with 8dfaba24bdce1ed525be63e98a5150377e2dcadaff451774ccd5a28234621c65 not found: ID does not exist"
Oct 30 12:45:54.611233 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.611186    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:45:54.620423 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.620396    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-4a92de-87c57c4f6-xrm72"]
Oct 30 12:45:54.910437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:54.910405    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" path="/var/lib/kubelet/pods/ec90dfa7-8dc3-4928-a13b-26571018407f/volumes"
Oct 30 12:45:57.251279 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:57.251219    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing" containerID="cri-o://18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2" gracePeriod=25
Oct 30 12:45:58.161855 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.161825    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 12:45:58.208196 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.208163    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 12:45:58.429792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.429764    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:45:58.566259 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.566225    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\") pod \"0e04b890-36bb-4f10-ac03-00b75d288c08\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") "
Oct 30 12:45:58.566473 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.566356    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\" (UniqueName: \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\") pod \"0e04b890-36bb-4f10-ac03-00b75d288c08\" (UID: \"0e04b890-36bb-4f10-ac03-00b75d288c08\") "
Oct 30 12:45:58.566663 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.566638    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume" pod "0e04b890-36bb-4f10-ac03-00b75d288c08" (UID: "0e04b890-36bb-4f10-ac03-00b75d288c08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:45:58.568532 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.568508    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv" pod "0e04b890-36bb-4f10-ac03-00b75d288c08" (UID: "0e04b890-36bb-4f10-ac03-00b75d288c08"). InnerVolumeSpecName "kube-api-access-bnqvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:45:58.590794 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.590763    4453 generic.go:356] "Generic (PLEG): container finished" podID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerID="f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b" exitCode=0
Oct 30 12:45:58.590917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.590809    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"
Oct 30 12:45:58.590917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.590858    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" event={"ID":"0e04b890-36bb-4f10-ac03-00b75d288c08","Type":"ContainerDied","Data":"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"}
Oct 30 12:45:58.590917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.590911    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8" event={"ID":"0e04b890-36bb-4f10-ac03-00b75d288c08","Type":"ContainerDied","Data":"68776148589e8644a2b8ff510e5af4941a59a8090d44fb32bfc20f41444c8460"}
Oct 30 12:45:58.591063 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.590940    4453 scope.go:122] "RemoveContainer" containerID="f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"
Oct 30 12:45:58.600230 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.600212    4453 scope.go:122] "RemoveContainer" containerID="f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"
Oct 30 12:45:58.600508 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:45:58.600490    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b\": container with ID starting with f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b not found: ID does not exist" containerID="f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"
Oct 30 12:45:58.600561 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.600515    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b"} err="failed to get container status \"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b\": rpc error: code = NotFound desc = could not find container \"f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b\": container with ID starting with f0cecf3ee8bdddbe4afbe5d67bb9c2bda3d92986f8c2411808b10ce04c80e61b not found: ID does not exist"
Oct 30 12:45:58.637140 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.637106    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:45:58.641091 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.641070    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4fff4f-786458586b-9k7k8"]
Oct 30 12:45:58.668212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.668159    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqvv\" (UniqueName: \"kubernetes.io/projected/0e04b890-36bb-4f10-ac03-00b75d288c08-kube-api-access-bnqvv\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:58.668212 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.668180    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e04b890-36bb-4f10-ac03-00b75d288c08-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:45:58.910493 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:45:58.910460    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" path="/var/lib/kubelet/pods/0e04b890-36bb-4f10-ac03-00b75d288c08/volumes"
Oct 30 12:46:02.577302 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:02.577256    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:03.162413 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:03.162371    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing" containerID="cri-o://b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957" gracePeriod=25
Oct 30 12:46:03.209226 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:03.209181    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing" containerID="cri-o://c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847" gracePeriod=25
Oct 30 12:46:07.379879 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.379829    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main" containerID="cri-o://1e3db5f38da1514a66432669b5b5eff37d9f28b19a984f5786c1c5ae4ddc21e7" gracePeriod=45
Oct 30 12:46:07.579316 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:07.579276    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:46:07.638601 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.638521    4453 generic.go:356] "Generic (PLEG): container finished" podID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerID="1e3db5f38da1514a66432669b5b5eff37d9f28b19a984f5786c1c5ae4ddc21e7" exitCode=0
Oct 30 12:46:07.638749 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.638601    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" event={"ID":"0208e127-6f17-4552-bd62-26cdaed45cf4","Type":"ContainerDied","Data":"1e3db5f38da1514a66432669b5b5eff37d9f28b19a984f5786c1c5ae4ddc21e7"}
Oct 30 12:46:07.676698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.676672    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 12:46:07.684817 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.684789    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\" (UniqueName: \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.684939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.684867    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.684939 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.684915    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.685041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.684970    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.685041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.684993    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.685041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685022    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\") pod \"0208e127-6f17-4552-bd62-26cdaed45cf4\" (UID: \"0208e127-6f17-4552-bd62-26cdaed45cf4\") "
Oct 30 12:46:07.685231 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685183    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.685320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685257    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.685447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685422    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.685570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685535    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.685570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685555    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.685570 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.685569    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.687702 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.687673    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:07.687816 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.687676    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "kube-api-access-sljvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:07.687988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.687959    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm" pod "0208e127-6f17-4552-bd62-26cdaed45cf4" (UID: "0208e127-6f17-4552-bd62-26cdaed45cf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:07.786322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.786289    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0208e127-6f17-4552-bd62-26cdaed45cf4-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.786322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.786318    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0208e127-6f17-4552-bd62-26cdaed45cf4-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:07.786322 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:07.786342    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljvr\" (UniqueName: \"kubernetes.io/projected/0208e127-6f17-4552-bd62-26cdaed45cf4-kube-api-access-sljvr\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.281990 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.281947    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" containerID="cri-o://8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5" gracePeriod=45
Oct 30 12:46:08.286407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.286365    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": EOF"
Oct 30 12:46:08.291041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.291008    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main" containerID="cri-o://5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0" gracePeriod=45
Oct 30 12:46:08.560478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.560451    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 12:46:08.594542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594511    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594607    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594701 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594669    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594698    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594746    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\" (UniqueName: \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594821 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594781    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\") pod \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\" (UID: \"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc\") "
Oct 30 12:46:08.594985 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.594840    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.595195 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.595168    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.595285 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.595270    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.595539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.595509    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.597629 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.597603    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.597820 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.597798    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:08.598048 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.598026    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b" pod "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" (UID: "a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc"). InnerVolumeSpecName "kube-api-access-v222b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:08.601751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.601729    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 12:46:08.650123 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.650089    4453 generic.go:356] "Generic (PLEG): container finished" podID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerID="5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0" exitCode=0
Oct 30 12:46:08.650360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.650145    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"
Oct 30 12:46:08.650360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.650186    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" event={"ID":"c8b26e16-f0aa-4d69-ab7e-6692ad20c506","Type":"ContainerDied","Data":"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"}
Oct 30 12:46:08.650360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.650223    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn" event={"ID":"c8b26e16-f0aa-4d69-ab7e-6692ad20c506","Type":"ContainerDied","Data":"8c6d693d4428e29679046526389593a499a8d157587c09dbc94e229d54117829"}
Oct 30 12:46:08.650360 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.650246    4453 scope.go:122] "RemoveContainer" containerID="5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"
Oct 30 12:46:08.652434 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.652402    4453 generic.go:356] "Generic (PLEG): container finished" podID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerID="8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5" exitCode=0
Oct 30 12:46:08.652744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.652460    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"
Oct 30 12:46:08.652744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.652530    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" event={"ID":"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc","Type":"ContainerDied","Data":"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"}
Oct 30 12:46:08.652744 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.652564    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq" event={"ID":"a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc","Type":"ContainerDied","Data":"6667484e0a8d4a8b400f5c497dc4276e588aff3780c7d5c0b3856bfb71a633ea"}
Oct 30 12:46:08.660319 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.660283    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj" event={"ID":"0208e127-6f17-4552-bd62-26cdaed45cf4","Type":"ContainerDied","Data":"a69e7ca0abbd65c340face52a5b79d48697dd72d83e6b59cbfe0c2929904380a"}
Oct 30 12:46:08.660558 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.660401    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"
Oct 30 12:46:08.665046 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.665020    4453 scope.go:122] "RemoveContainer" containerID="5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"
Oct 30 12:46:08.665406 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:08.665377    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0\": container with ID starting with 5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0 not found: ID does not exist" containerID="5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"
Oct 30 12:46:08.665522 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.665425    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0"} err="failed to get container status \"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0\": rpc error: code = NotFound desc = could not find container \"5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0\": container with ID starting with 5dfe5f159e2812c5a21aa18f389d01ebac08503344cfa1669d8c4f6e1eedc9d0 not found: ID does not exist"
Oct 30 12:46:08.665522 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.665450    4453 scope.go:122] "RemoveContainer" containerID="8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"
Oct 30 12:46:08.680437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.680418    4453 scope.go:122] "RemoveContainer" containerID="8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"
Oct 30 12:46:08.680719 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:08.680698    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5\": container with ID starting with 8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5 not found: ID does not exist" containerID="8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"
Oct 30 12:46:08.680822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.680726    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5"} err="failed to get container status \"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5\": rpc error: code = NotFound desc = could not find container \"8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5\": container with ID starting with 8dcbd6143357b70a597b4412c721709cb5c098c21dfd468c3e502d4a05849ea5 not found: ID does not exist"
Oct 30 12:46:08.680822 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.680745    4453 scope.go:122] "RemoveContainer" containerID="1e3db5f38da1514a66432669b5b5eff37d9f28b19a984f5786c1c5ae4ddc21e7"
Oct 30 12:46:08.695978 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.695939    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696000    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696055    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696097    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696160    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\" (UniqueName: \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696277    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\") pod \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\" (UID: \"c8b26e16-f0aa-4d69-ab7e-6692ad20c506\") "
Oct 30 12:46:08.696790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696761    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.696912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696800    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.696912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696826    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.696912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696843    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.696912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696856    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v222b\" (UniqueName: \"kubernetes.io/projected/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-kube-api-access-v222b\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.696912 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.696869    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.697198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.697055    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.697198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.697102    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.699898 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.699854    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "kube-api-access-hnr8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:08.700027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.700007    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:08.700522 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.700501    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs" pod "c8b26e16-f0aa-4d69-ab7e-6692ad20c506" (UID: "c8b26e16-f0aa-4d69-ab7e-6692ad20c506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:08.720147 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.720119    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 12:46:08.725653 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.725628    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-1d9b1a/distinct-2-w2-1d9b1a-kserve-54d96564-8hpxj"]
Oct 30 12:46:08.755037 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.755005    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 12:46:08.769131 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.768322    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/distinct-2-w0-3ccdab-kserve-7d9446f4b8-pppqq"]
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798087    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798118    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798127    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798135    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798143    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.798163 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.798152    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnr8k\" (UniqueName: \"kubernetes.io/projected/c8b26e16-f0aa-4d69-ab7e-6692ad20c506-kube-api-access-hnr8k\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:08.911461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.911427    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" path="/var/lib/kubelet/pods/0208e127-6f17-4552-bd62-26cdaed45cf4/volumes"
Oct 30 12:46:08.911883 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.911869    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" path="/var/lib/kubelet/pods/a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc/volumes"
Oct 30 12:46:08.985640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.985609    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 12:46:08.990833 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:08.990804    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3ccdab/premium-w0-3ccdab-kserve-796777c4b4-9h4rn"]
Oct 30 12:46:10.912641 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:10.912606    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" path="/var/lib/kubelet/pods/c8b26e16-f0aa-4d69-ab7e-6692ad20c506/volumes"
Oct 30 12:46:11.050751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:11.050712    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:11.050920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:11.050811    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:12.577107 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:12.577060    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:12.855766 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:12.855682    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 12:46:13.219619 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.219581    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 12:46:13.219885 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.219854    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" podUID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" containerName="maas-api" containerID="cri-o://b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7" gracePeriod=30
Oct 30 12:46:13.487416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.487388    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 12:46:13.566635 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.566608    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\") pod \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") "
Oct 30 12:46:13.566804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.566678    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\") pod \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") "
Oct 30 12:46:13.566804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.566782    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\" (UniqueName: \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\") pod \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") "
Oct 30 12:46:13.566902 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.566817    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\") pod \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\" (UID: \"a3bffa8b-cc2a-4829-9615-b92ba71052c1\") "
Oct 30 12:46:13.567287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.567253    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca" pod "a3bffa8b-cc2a-4829-9615-b92ba71052c1" (UID: "a3bffa8b-cc2a-4829-9615-b92ba71052c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:13.568818 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.568788    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls" pod "a3bffa8b-cc2a-4829-9615-b92ba71052c1" (UID: "a3bffa8b-cc2a-4829-9615-b92ba71052c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:13.568931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.568832    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs" pod "a3bffa8b-cc2a-4829-9615-b92ba71052c1" (UID: "a3bffa8b-cc2a-4829-9615-b92ba71052c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:13.568931 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.568887    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6" pod "a3bffa8b-cc2a-4829-9615-b92ba71052c1" (UID: "a3bffa8b-cc2a-4829-9615-b92ba71052c1"). InnerVolumeSpecName "kube-api-access-bf7p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:13.581853 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.581826    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 12:46:13.582179 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.582044    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" podUID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" containerName="maas-api" containerID="cri-o://54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15" gracePeriod=30
Oct 30 12:46:13.668202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.668172    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3bffa8b-cc2a-4829-9615-b92ba71052c1-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.668202 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.668201    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.668416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.668213    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3bffa8b-cc2a-4829-9615-b92ba71052c1-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.668416 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.668225    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf7p6\" (UniqueName: \"kubernetes.io/projected/a3bffa8b-cc2a-4829-9615-b92ba71052c1-kube-api-access-bf7p6\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.694841 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.694807    4453 generic.go:356] "Generic (PLEG): container finished" podID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" containerID="b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7" exitCode=0
Oct 30 12:46:13.695011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.694856    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"
Oct 30 12:46:13.695011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.694898    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" event={"ID":"a3bffa8b-cc2a-4829-9615-b92ba71052c1","Type":"ContainerDied","Data":"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"}
Oct 30 12:46:13.695011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.694968    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f" event={"ID":"a3bffa8b-cc2a-4829-9615-b92ba71052c1","Type":"ContainerDied","Data":"87131a3b6eb77020a53c1f14f58d537ccfc173b0488fae2b10804c84656293ca"}
Oct 30 12:46:13.695011 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.695006    4453 scope.go:122] "RemoveContainer" containerID="b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"
Oct 30 12:46:13.709116 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.709090    4453 scope.go:122] "RemoveContainer" containerID="b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"
Oct 30 12:46:13.709559 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:13.709509    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7\": container with ID starting with b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7 not found: ID does not exist" containerID="b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"
Oct 30 12:46:13.709723 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.709568    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7"} err="failed to get container status \"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7\": rpc error: code = NotFound desc = could not find container \"b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7\": container with ID starting with b3da6a8da20e39df75046253c6665c0599c722eef441f5de5cd1a01dc9d33ed7 not found: ID does not exist"
Oct 30 12:46:13.744193 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.744107    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 12:46:13.753685 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.753657    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3ccdab-69d675dd58-c997f"]
Oct 30 12:46:13.828804 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.828780    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 12:46:13.869895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.869852    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\") pod \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") "
Oct 30 12:46:13.870054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.869912    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\") pod \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") "
Oct 30 12:46:13.870054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.869999    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\") pod \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") "
Oct 30 12:46:13.870054 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.870021    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\" (UniqueName: \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\") pod \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\" (UID: \"584d9ada-9105-47cd-b5a1-6ee303a7b3cf\") "
Oct 30 12:46:13.870288 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.870259    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca" pod "584d9ada-9105-47cd-b5a1-6ee303a7b3cf" (UID: "584d9ada-9105-47cd-b5a1-6ee303a7b3cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:13.870513 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.870493    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.872107 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.872083    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls" pod "584d9ada-9105-47cd-b5a1-6ee303a7b3cf" (UID: "584d9ada-9105-47cd-b5a1-6ee303a7b3cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:13.872243 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.872221    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl" pod "584d9ada-9105-47cd-b5a1-6ee303a7b3cf" (UID: "584d9ada-9105-47cd-b5a1-6ee303a7b3cf"). InnerVolumeSpecName "kube-api-access-p54rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:13.872320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.872268    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs" pod "584d9ada-9105-47cd-b5a1-6ee303a7b3cf" (UID: "584d9ada-9105-47cd-b5a1-6ee303a7b3cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:13.971539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.971508    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.971539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.971533    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:13.971539 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:13.971543    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54rl\" (UniqueName: \"kubernetes.io/projected/584d9ada-9105-47cd-b5a1-6ee303a7b3cf-kube-api-access-p54rl\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:14.701665 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.701628    4453 generic.go:356] "Generic (PLEG): container finished" podID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" containerID="54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15" exitCode=0
Oct 30 12:46:14.702119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.701679    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"
Oct 30 12:46:14.702119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.701707    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" event={"ID":"584d9ada-9105-47cd-b5a1-6ee303a7b3cf","Type":"ContainerDied","Data":"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"}
Oct 30 12:46:14.702119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.701756    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l" event={"ID":"584d9ada-9105-47cd-b5a1-6ee303a7b3cf","Type":"ContainerDied","Data":"13dd9a520d7db03c0ae0d746627bfedc795de1d5909beacdc5a2c912b5a82ebe"}
Oct 30 12:46:14.702119 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.701777    4453 scope.go:122] "RemoveContainer" containerID="54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"
Oct 30 12:46:14.711158 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.711142    4453 scope.go:122] "RemoveContainer" containerID="54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"
Oct 30 12:46:14.711421 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:14.711400    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15\": container with ID starting with 54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15 not found: ID does not exist" containerID="54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"
Oct 30 12:46:14.711474 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.711427    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15"} err="failed to get container status \"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15\": rpc error: code = NotFound desc = could not find container \"54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15\": container with ID starting with 54516f58d6c327eb98e8e47dadd8897eadbfae1d8c323fddb4f574b60b7f3f15 not found: ID does not exist"
Oct 30 12:46:14.744523 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.744501    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 12:46:14.753946 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.753925    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1d9b1a-56c574d6b6-5pt7l"]
Oct 30 12:46:14.910378 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.910324    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" path="/var/lib/kubelet/pods/584d9ada-9105-47cd-b5a1-6ee303a7b3cf/volumes"
Oct 30 12:46:14.910867 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:14.910844    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" path="/var/lib/kubelet/pods/a3bffa8b-cc2a-4829-9615-b92ba71052c1/volumes"
Oct 30 12:46:17.520394 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.520320    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:46:17.608696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.608666    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\") pod \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") "
Oct 30 12:46:17.608849 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.608766    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\" (UniqueName: \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\") pod \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\" (UID: \"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9\") "
Oct 30 12:46:17.609117 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.609092    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume" pod "cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" (UID: "cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:17.610878 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.610852    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs" pod "cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" (UID: "cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9"). InnerVolumeSpecName "kube-api-access-dcvrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:17.710437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.710402    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:17.710437 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.710431    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvrs\" (UniqueName: \"kubernetes.io/projected/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9-kube-api-access-dcvrs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:17.718640 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.718610    4453 generic.go:356] "Generic (PLEG): container finished" podID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerID="18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2" exitCode=0
Oct 30 12:46:17.718783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.718653    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"
Oct 30 12:46:17.718783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.718692    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" event={"ID":"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9","Type":"ContainerDied","Data":"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"}
Oct 30 12:46:17.718783 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.718765    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6" event={"ID":"cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9","Type":"ContainerDied","Data":"06b8ee30e524765e344ed8a441a96c7638a3fa4a3a806a809ad0591ce061481a"}
Oct 30 12:46:17.718940 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.718787    4453 scope.go:122] "RemoveContainer" containerID="18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"
Oct 30 12:46:17.728249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.728232    4453 scope.go:122] "RemoveContainer" containerID="18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"
Oct 30 12:46:17.728564 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:17.728538    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2\": container with ID starting with 18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2 not found: ID does not exist" containerID="18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"
Oct 30 12:46:17.728625 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.728581    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2"} err="failed to get container status \"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2\": rpc error: code = NotFound desc = could not find container \"18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2\": container with ID starting with 18567f149572e6e6382f481048ef4bae024ed978681897273afcc3bca5d2c3f2 not found: ID does not exist"
Oct 30 12:46:17.761209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.761184    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:46:17.766810 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:17.766788    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2bc4e2-6c55d57495-vc2n6"]
Oct 30 12:46:18.910431 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:18.910395    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" path="/var/lib/kubelet/pods/cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9/volumes"
Oct 30 12:46:21.051020 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:21.050988    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:21.051407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:21.050985    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 12:46:22.314361 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:22.312172    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 12:46:22.315534 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:22.315498    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" containerID="cri-o://a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714" gracePeriod=30
Oct 30 12:46:23.464457 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.464425    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 12:46:23.491413 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.491374    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 12:46:23.572645 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.572621    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\") pod \"44dc9bf9-0842-464f-a70c-e1328f61764f\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") "
Oct 30 12:46:23.572792 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.572692    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\" (UniqueName: \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\") pod \"44dc9bf9-0842-464f-a70c-e1328f61764f\" (UID: \"44dc9bf9-0842-464f-a70c-e1328f61764f\") "
Oct 30 12:46:23.572838 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.572807    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\") pod \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") "
Oct 30 12:46:23.572895 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.572844    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\" (UniqueName: \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\") pod \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\" (UID: \"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274\") "
Oct 30 12:46:23.573177 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.573148    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume" pod "44dc9bf9-0842-464f-a70c-e1328f61764f" (UID: "44dc9bf9-0842-464f-a70c-e1328f61764f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:23.573347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.573181    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume" pod "d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" (UID: "d72e6d7c-cd0f-4e7b-89a3-7894d03f2274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:23.573347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.573301    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.573347 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.573320    4453 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dc9bf9-0842-464f-a70c-e1328f61764f-plugins-config-volume\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.574858 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.574830    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft" pod "44dc9bf9-0842-464f-a70c-e1328f61764f" (UID: "44dc9bf9-0842-464f-a70c-e1328f61764f"). InnerVolumeSpecName "kube-api-access-sgwft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:23.574943 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.574854    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk" pod "d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" (UID: "d72e6d7c-cd0f-4e7b-89a3-7894d03f2274"). InnerVolumeSpecName "kube-api-access-nrpnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:23.674092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.674067    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpnk\" (UniqueName: \"kubernetes.io/projected/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274-kube-api-access-nrpnk\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.674092 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.674088    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgwft\" (UniqueName: \"kubernetes.io/projected/44dc9bf9-0842-464f-a70c-e1328f61764f-kube-api-access-sgwft\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.751055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.750994    4453 generic.go:356] "Generic (PLEG): container finished" podID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerID="b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957" exitCode=0
Oct 30 12:46:23.751055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.751039    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"
Oct 30 12:46:23.751220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.751075    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" event={"ID":"44dc9bf9-0842-464f-a70c-e1328f61764f","Type":"ContainerDied","Data":"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"}
Oct 30 12:46:23.751220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.751123    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk" event={"ID":"44dc9bf9-0842-464f-a70c-e1328f61764f","Type":"ContainerDied","Data":"2acbb6d6349990f12e04105913e093c37ccb264d070642ba7ca645b9c1878b78"}
Oct 30 12:46:23.751220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.751146    4453 scope.go:122] "RemoveContainer" containerID="b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"
Oct 30 12:46:23.752426 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.752396    4453 generic.go:356] "Generic (PLEG): container finished" podID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerID="c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847" exitCode=0
Oct 30 12:46:23.752542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.752436    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"
Oct 30 12:46:23.752542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.752441    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" event={"ID":"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274","Type":"ContainerDied","Data":"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"}
Oct 30 12:46:23.752542 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.752487    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh" event={"ID":"d72e6d7c-cd0f-4e7b-89a3-7894d03f2274","Type":"ContainerDied","Data":"e9b68286a6386dd7517ea5a4c20d857b91309936f0e9369e0defdc85e59bf44b"}
Oct 30 12:46:23.760167 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.760049    4453 scope.go:122] "RemoveContainer" containerID="b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"
Oct 30 12:46:23.760353 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:23.760301    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957\": container with ID starting with b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957 not found: ID does not exist" containerID="b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"
Oct 30 12:46:23.760442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.760354    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957"} err="failed to get container status \"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957\": rpc error: code = NotFound desc = could not find container \"b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957\": container with ID starting with b3a4b654bc1f93b61030a99d7cc712895a2e8d3ec1012434170add26f69a5957 not found: ID does not exist"
Oct 30 12:46:23.760442 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.760379    4453 scope.go:122] "RemoveContainer" containerID="c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"
Oct 30 12:46:23.768252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.768235    4453 scope.go:122] "RemoveContainer" containerID="c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"
Oct 30 12:46:23.768520 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:23.768502    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847\": container with ID starting with c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847 not found: ID does not exist" containerID="c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"
Oct 30 12:46:23.768574 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.768526    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847"} err="failed to get container status \"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847\": rpc error: code = NotFound desc = could not find container \"c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847\": container with ID starting with c8f0f870318907a8a09e5b518987e9422fce7d32dcb09d14c53a53567a61b847 not found: ID does not exist"
Oct 30 12:46:23.793696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.793674    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 12:46:23.803070 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.803049    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-1d9b1a-69d786c854-5s7wk"]
Oct 30 12:46:23.831411 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.831390    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 12:46:23.837209 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:23.837188    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1d9b1a-7d97888b9f-wgvkh"]
Oct 30 12:46:24.909449 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:24.909406    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" path="/var/lib/kubelet/pods/44dc9bf9-0842-464f-a70c-e1328f61764f/volumes"
Oct 30 12:46:24.909987 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:24.909963    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" path="/var/lib/kubelet/pods/d72e6d7c-cd0f-4e7b-89a3-7894d03f2274/volumes"
Oct 30 12:46:26.440743 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:26.440704    4453 patch_prober.go:29] interesting pod/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:46:26.441112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:26.440770    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:27.570525 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.570501    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 12:46:27.711927 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.711893    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.711933    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.711964    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712094 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712080    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712247 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712117    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712247 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712196    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712247 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712219    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.712453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712281    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712313    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712453 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712381    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\") pod \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\" (UID: \"fe166ae9-5782-4efe-81e0-ac9d359e1c33\") "
Oct 30 12:46:27.712617 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712467    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.712703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712681    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.712770 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712732    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.712865 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712842    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.712935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712867    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.712935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712891    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.712935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712907    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.712935 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.712921    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.714349 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.714298    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "kube-api-access-pfnqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:27.714461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.714351    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:27.714461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.714390    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.714461 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.714429    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo" pod "fe166ae9-5782-4efe-81e0-ac9d359e1c33" (UID: "fe166ae9-5782-4efe-81e0-ac9d359e1c33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:27.784116 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.784080    4453 generic.go:356] "Generic (PLEG): container finished" podID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerID="a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714" exitCode=0
Oct 30 12:46:27.784287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.784138    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"
Oct 30 12:46:27.784287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.784176    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" event={"ID":"fe166ae9-5782-4efe-81e0-ac9d359e1c33","Type":"ContainerDied","Data":"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"}
Oct 30 12:46:27.784287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.784236    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw" event={"ID":"fe166ae9-5782-4efe-81e0-ac9d359e1c33","Type":"ContainerDied","Data":"488cf60518866dc4b16bb77acca4fadca42f186912baba15ea38c5373e6a80b6"}
Oct 30 12:46:27.784287 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.784261    4453 scope.go:122] "RemoveContainer" containerID="a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"
Oct 30 12:46:27.793563 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.793547    4453 scope.go:122] "RemoveContainer" containerID="a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"
Oct 30 12:46:27.793824 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:27.793806    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714\": container with ID starting with a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714 not found: ID does not exist" containerID="a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"
Oct 30 12:46:27.793910 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.793832    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714"} err="failed to get container status \"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714\": rpc error: code = NotFound desc = could not find container \"a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714\": container with ID starting with a03e1ea6ec62c6acb15a416c7f3edd9d1d201889dfca804d370ca2fcd2a08714 not found: ID does not exist"
Oct 30 12:46:27.814256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.814231    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.814256 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.814253    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.814412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.814261    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.814412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.814270    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnqh\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-kube-api-access-pfnqh\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.814412 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.814280    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe166ae9-5782-4efe-81e0-ac9d359e1c33-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:27.828643 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.828615    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 12:46:27.832697 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.832634    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1d9b1a-openshift-default-59d77cb49c-xw9tw"]
Oct 30 12:46:27.876027 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:27.875993    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main" containerID="cri-o://a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23" gracePeriod=45
Oct 30 12:46:28.115664 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.115640    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 12:46:28.217750 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.217712    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.217909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.217764    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\" (UniqueName: \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.217909 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.217800    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.218022 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.217957    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.218086 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218047    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.218303 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218171    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\") pod \"99db1f4a-8b87-4036-acca-bb160b1580d2\" (UID: \"99db1f4a-8b87-4036-acca-bb160b1580d2\") "
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218404    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218480    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218490    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218633    4453 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-model-cache\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218654    4453 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-home\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.218698 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.218667    4453 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-tmp-dir\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.220719 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.220691    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:28.220823 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.220787    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "kube-api-access-vm5rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:28.220884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.220865    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm" pod "99db1f4a-8b87-4036-acca-bb160b1580d2" (UID: "99db1f4a-8b87-4036-acca-bb160b1580d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:28.320216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.320127    4453 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99db1f4a-8b87-4036-acca-bb160b1580d2-dshm\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.320216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.320159    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm5rl\" (UniqueName: \"kubernetes.io/projected/99db1f4a-8b87-4036-acca-bb160b1580d2-kube-api-access-vm5rl\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.320216 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.320168    4453 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99db1f4a-8b87-4036-acca-bb160b1580d2-tls-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.790500 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.790463    4453 generic.go:356] "Generic (PLEG): container finished" podID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerID="a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23" exitCode=0
Oct 30 12:46:28.790955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.790512    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"
Oct 30 12:46:28.790955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.790551    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" event={"ID":"99db1f4a-8b87-4036-acca-bb160b1580d2","Type":"ContainerDied","Data":"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"}
Oct 30 12:46:28.790955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.790593    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq" event={"ID":"99db1f4a-8b87-4036-acca-bb160b1580d2","Type":"ContainerDied","Data":"4771183838ac210fd6bb144f7e580d79f9bdce79e552cd142f980440f390412d"}
Oct 30 12:46:28.790955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.790613    4453 scope.go:122] "RemoveContainer" containerID="a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"
Oct 30 12:46:28.799551 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.799533    4453 scope.go:122] "RemoveContainer" containerID="a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"
Oct 30 12:46:28.799816 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:28.799800    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23\": container with ID starting with a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23 not found: ID does not exist" containerID="a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"
Oct 30 12:46:28.799884 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.799825    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23"} err="failed to get container status \"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23\": rpc error: code = NotFound desc = could not find container \"a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23\": container with ID starting with a776d865ddd3b3f098093afb95b0f644309fc573ff4dbb7f70b0477b99afbf23 not found: ID does not exist"
Oct 30 12:46:28.834228 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.834195    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 12:46:28.838357 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.838309    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e43a66/premium-w1-e43a66-kserve-5bdb95c45d-lnjnq"]
Oct 30 12:46:28.910396 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.910365    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" path="/var/lib/kubelet/pods/99db1f4a-8b87-4036-acca-bb160b1580d2/volumes"
Oct 30 12:46:28.910846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:28.910833    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" path="/var/lib/kubelet/pods/fe166ae9-5782-4efe-81e0-ac9d359e1c33/volumes"
Oct 30 12:46:34.595406 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.595366    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 12:46:34.596055 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.595581    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" podUID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" containerName="maas-api" containerID="cri-o://70153e3d40ac3776408e8e80fbb752e685910f99c2a5c3811fe69d00c820180d" gracePeriod=30
Oct 30 12:46:34.824903 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.824869    4453 generic.go:356] "Generic (PLEG): container finished" podID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" containerID="70153e3d40ac3776408e8e80fbb752e685910f99c2a5c3811fe69d00c820180d" exitCode=0
Oct 30 12:46:34.825077 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.824940    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" event={"ID":"818bca10-8dbd-4d59-98c6-4b60b95f0ae2","Type":"ContainerDied","Data":"70153e3d40ac3776408e8e80fbb752e685910f99c2a5c3811fe69d00c820180d"}
Oct 30 12:46:34.865458 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.865430    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 12:46:34.995868 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.995835    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\") pod \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") "
Oct 30 12:46:34.996041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.995895    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\" (UniqueName: \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\") pod \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") "
Oct 30 12:46:34.996041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.995919    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\") pod \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") "
Oct 30 12:46:34.996041 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.995963    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") pod \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\" (UID: \"818bca10-8dbd-4d59-98c6-4b60b95f0ae2\") "
Oct 30 12:46:34.996229 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.996208    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca" pod "818bca10-8dbd-4d59-98c6-4b60b95f0ae2" (UID: "818bca10-8dbd-4d59-98c6-4b60b95f0ae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:34.998107 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.998077    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn" pod "818bca10-8dbd-4d59-98c6-4b60b95f0ae2" (UID: "818bca10-8dbd-4d59-98c6-4b60b95f0ae2"). InnerVolumeSpecName "kube-api-access-wlmgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:34.998257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.998200    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs" pod "818bca10-8dbd-4d59-98c6-4b60b95f0ae2" (UID: "818bca10-8dbd-4d59-98c6-4b60b95f0ae2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:34.998257 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:34.998213    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls" pod "818bca10-8dbd-4d59-98c6-4b60b95f0ae2" (UID: "818bca10-8dbd-4d59-98c6-4b60b95f0ae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:35.097153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.097103    4453 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-openshift-service-ca\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:35.097153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.097135    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlmgn\" (UniqueName: \"kubernetes.io/projected/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-kube-api-access-wlmgn\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:35.097153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.097145    4453 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-metrics-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:35.097153 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.097154    4453 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818bca10-8dbd-4d59-98c6-4b60b95f0ae2-maas-api-tls\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:35.831555 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.831521    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"
Oct 30 12:46:35.832093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.831522    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4" event={"ID":"818bca10-8dbd-4d59-98c6-4b60b95f0ae2","Type":"ContainerDied","Data":"e70ab6e15dd0d898c893112abed511ccb13125abf5407a5338d5aca96b103912"}
Oct 30 12:46:35.832093 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.831823    4453 scope.go:122] "RemoveContainer" containerID="70153e3d40ac3776408e8e80fbb752e685910f99c2a5c3811fe69d00c820180d"
Oct 30 12:46:35.875580 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.875553    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 12:46:35.884751 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:35.884716    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e43a66-568846dfd8-2b5x4"]
Oct 30 12:46:36.909718 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:36.909682    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" path="/var/lib/kubelet/pods/818bca10-8dbd-4d59-98c6-4b60b95f0ae2/volumes"
Oct 30 12:46:37.570733 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:37.570694    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:46:42.447200 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:42.447160    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 12:46:42.447749 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:42.447638    4453 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" containerID="cri-o://30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a" gracePeriod=30
Oct 30 12:46:43.441982 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:43.441950    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d59685b8f-lbdsm_593c626b-4093-4bc3-8b00-08c8a8c6af8d/authorino/0.log"
Oct 30 12:46:45.836114 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:45.836082    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:46:46.755524 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:46.755486    4453 patch_prober.go:29] interesting pod/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 12:46:46.755739 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:46.755620    4453 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 12:46:47.322395 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.322368    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:46:47.690128 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.690099    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 12:46:47.724857 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724821    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724869    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724893    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724920    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724960    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725002 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.724982    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725015    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725063    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725117    4453 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\") pod \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\" (UID: \"ebdc5f2a-7e14-473b-b459-8b78e39b0be8\") "
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725170    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725217    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:47.725295 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725263    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:47.725660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725513    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:47.725660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725567    4453 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istiod-ca-cert\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.725660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725588    4453 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-certs\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.725660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725601    4453 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-data\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.725660 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.725601    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:47.727651 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.727631    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:47.727746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.727730    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:47.727746 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.727732    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "kube-api-access-4qnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:47.727875 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.727750    4453 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo" pod "ebdc5f2a-7e14-473b-b459-8b78e39b0be8" (UID: "ebdc5f2a-7e14-473b-b459-8b78e39b0be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 12:46:47.826455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826425    4453 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-token\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.826455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826450    4453 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-credential-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.826455 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826461    4453 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-envoy\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.826659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826471    4453 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-istio-podinfo\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.826659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826478    4453 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-workload-socket\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.826659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.826486    4453 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qnb2\" (UniqueName: \"kubernetes.io/projected/ebdc5f2a-7e14-473b-b459-8b78e39b0be8-kube-api-access-4qnb2\") on node \"ip-10-0-143-18.ec2.internal\" DevicePath \"\""
Oct 30 12:46:47.894486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.894414    4453 generic.go:356] "Generic (PLEG): container finished" podID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerID="30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a" exitCode=0
Oct 30 12:46:47.894486 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.894471    4453 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"
Oct 30 12:46:47.894675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.894502    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" event={"ID":"ebdc5f2a-7e14-473b-b459-8b78e39b0be8","Type":"ContainerDied","Data":"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"}
Oct 30 12:46:47.894675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.894598    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv" event={"ID":"ebdc5f2a-7e14-473b-b459-8b78e39b0be8","Type":"ContainerDied","Data":"08283dfa91edff31f6f7b9d707c3c90d00434c359ddc5a5a2c1f69a566a28076"}
Oct 30 12:46:47.894675 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.894631    4453 scope.go:122] "RemoveContainer" containerID="30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"
Oct 30 12:46:47.903737 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.903718    4453 scope.go:122] "RemoveContainer" containerID="30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"
Oct 30 12:46:47.904005 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:46:47.903987    4453 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a\": container with ID starting with 30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a not found: ID does not exist" containerID="30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"
Oct 30 12:46:47.904057 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.904012    4453 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a"} err="failed to get container status \"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a\": rpc error: code = NotFound desc = could not find container \"30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a\": container with ID starting with 30f926014a9ab6d877258f38dcc3fb99528609cd7341ed17d93f35f8e995775a not found: ID does not exist"
Oct 30 12:46:47.938657 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.938635    4453 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 12:46:47.943076 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:47.943053    4453 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e43a66-openshift-default-7d59bc8989-kgncv"]
Oct 30 12:46:48.319988 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:48.319916    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d59685b8f-lbdsm_593c626b-4093-4bc3-8b00-08c8a8c6af8d/authorino/0.log"
Oct 30 12:46:48.909696 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:48.909656    4453 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" path="/var/lib/kubelet/pods/ebdc5f2a-7e14-473b-b459-8b78e39b0be8/volumes"
Oct 30 12:46:59.054073 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:46:59.054043    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:00.572252 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:00.572219    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:01.559722 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:01.559683    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c6d5fb647-8rlbz_c6b9e879-cbef-433f-8388-15fc39e8b78c/maas-api/0.log"
Oct 30 12:47:06.290862 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:06.290823    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d59685b8f-lbdsm_593c626b-4093-4bc3-8b00-08c8a8c6af8d/authorino/0.log"
Oct 30 12:47:07.569549 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:47:07.569513    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:47:08.760628 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:08.760597    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7htsq_32e04985-dc6f-43e4-ba67-2ae07d896050/discovery/0.log"
Oct 30 12:47:08.975749 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:08.975718    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-p6rqz_f404404b-d627-461a-b827-1e4e5bafa0fc/kube-auth-proxy/0.log"
Oct 30 12:47:09.330930 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:09.330902    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hpp9t_c5ef9f10-6825-4798-9579-2d1f3fed2dba/istio-proxy/0.log"
Oct 30 12:47:09.894703 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:09.894669    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-z7kbc_ec9cb544-80ec-428b-b007-bff354808519/istio-proxy/0.log"
Oct 30 12:47:10.292161 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:10.292088    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kfvvk_2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674/payload-pre-processing/0.log"
Oct 30 12:47:10.484777 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:10.484734    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 12:47:11.465598 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:11.465568    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-cdwnb_7d6a9d3d-d4f6-4bd6-acab-c305192ca954/main/0.log"
Oct 30 12:47:11.649780 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:11.649752    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kzmlb_3fab727a-0746-46f5-8bf8-d230acc8ee08/main/0.log"
Oct 30 12:47:11.831407 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:11.831308    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-f9zsw_cd9e6fb6-50e7-43b1-b99b-51bf284196df/main/0.log"
Oct 30 12:47:12.031915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:12.031884    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-mz9p2_9363a866-fbd8-469d-939c-44211df30df7/main/0.log"
Oct 30 12:47:12.590249 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:12.590213    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-99czt_fd3d1ac0-2e49-4c83-9c5b-f09a45326725/main/0.log"
Oct 30 12:47:12.797370 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:12.797324    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-l5z9f_7843c47a-0b1d-4004-a852-923b462f0c5c/main/0.log"
Oct 30 12:47:29.211012 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:29.210913    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hgkvl_2123026e-ac5a-4ef1-80f6-76b1bcc174c1/global-pull-secret-syncer/0.log"
Oct 30 12:47:30.048376 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:30.048345    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6d7hf_44f9925c-7d11-4296-9593-7a747f8d5916/konnectivity-agent/0.log"
Oct 30 12:47:30.611044 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:30.611009    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-18.ec2.internal_6905fd60ff57bade2ab9f0792ffa70c2/haproxy/0.log"
Oct 30 12:47:37.178612 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:37.178578    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d59685b8f-lbdsm_593c626b-4093-4bc3-8b00-08c8a8c6af8d/authorino/0.log"
Oct 30 12:47:37.571754 ip-10-0-143-18 kubenswrapper[4453]: E1030 12:47:37.571631    4453 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv5lx" containerName="extractor"
Oct 30 12:47:37.579920 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:37.579896    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:39.080254 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:39.080221    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:40.719302 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719262    4453 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"]
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719772    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719791    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719806    4453 cpu_manager.go:414] "containerMap: removing container" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing"
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719813    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing"
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719829    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" containerName="revoke-keys"
Oct 30 12:47:40.719840 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719837    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" containerName="revoke-keys"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719850    4453 cpu_manager.go:414] "containerMap: removing container" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719856    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719877    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719883    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719892    4453 cpu_manager.go:414] "containerMap: removing container" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719899    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719912    4453 cpu_manager.go:414] "containerMap: removing container" podUID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" containerName="maas-api"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719920    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" containerName="maas-api"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719928    4453 cpu_manager.go:414] "containerMap: removing container" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719935    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719952    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719961    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719971    4453 cpu_manager.go:414] "containerMap: removing container" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719978    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719988    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.719996    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720006    4453 cpu_manager.go:414] "containerMap: removing container" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720013    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720022    4453 cpu_manager.go:414] "containerMap: removing container" podUID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" containerName="maas-api"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720029    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" containerName="maas-api"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720056    4453 cpu_manager.go:414] "containerMap: removing container" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720065    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720076    4453 cpu_manager.go:414] "containerMap: removing container" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720084    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main"
Oct 30 12:47:40.720088 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720099    4453 cpu_manager.go:414] "containerMap: removing container" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720108    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720118    4453 cpu_manager.go:414] "containerMap: removing container" podUID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" containerName="maas-api"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720125    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" containerName="maas-api"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720135    4453 cpu_manager.go:414] "containerMap: removing container" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720142    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720159    4453 cpu_manager.go:414] "containerMap: removing container" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720167    4453 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720252    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0208e127-6f17-4552-bd62-26cdaed45cf4" containerName="main"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720260    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720267    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe166ae9-5782-4efe-81e0-ac9d359e1c33" containerName="istio-proxy"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720275    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec90dfa7-8dc3-4928-a13b-26571018407f" containerName="payload-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720284    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720290    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="818bca10-8dbd-4d59-98c6-4b60b95f0ae2" containerName="maas-api"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720300    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e680947-7fdd-4f28-a956-148150b3c1ac" containerName="cleanup"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720304    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="99db1f4a-8b87-4036-acca-bb160b1580d2" containerName="main"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720311    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72e6d7c-cd0f-4e7b-89a3-7894d03f2274" containerName="payload-pre-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720316    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="584d9ada-9105-47cd-b5a1-6ee303a7b3cf" containerName="maas-api"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720321    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdc5f2a-7e14-473b-b459-8b78e39b0be8" containerName="istio-proxy"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720326    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e04b890-36bb-4f10-ac03-00b75d288c08" containerName="payload-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720344    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="44dc9bf9-0842-464f-a70c-e1328f61764f" containerName="payload-processing"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720350    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5afb0c-baac-474b-8c1e-a8aa8f52b67a" containerName="revoke-keys"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720355    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b26e16-f0aa-4d69-ab7e-6692ad20c506" containerName="main"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720360    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f5ef93-6f28-4ed2-a7a4-148edbb9f9fc" containerName="main"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720365    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="01281026-c67e-4b92-baa9-fe0215e1a027" containerName="cleanup"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720371    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3bffa8b-cc2a-4829-9615-b92ba71052c1" containerName="maas-api"
Oct 30 12:47:40.721096 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.720376    4453 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb8e5aad-5b8b-40b3-b073-fcb9d09ebcd9" containerName="payload-pre-processing"
Oct 30 12:47:40.723624 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.723601    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.726678 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.726652    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:40.735536 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.735516    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"kube-root-ca.crt\""
Oct 30 12:47:40.735795 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.735778    4453 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c97ck\"/\"default-dockercfg-nn6n8\""
Oct 30 12:47:40.745544 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.745515    4453 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c97ck\"/\"openshift-service-ca.crt\""
Oct 30 12:47:40.748144 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.748099    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"]
Oct 30 12:47:40.815915 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.815877    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkqh\" (UniqueName: \"kubernetes.io/projected/2ca3b561-90f1-4c32-8ed9-40bb1e219918-kube-api-access-4vkqh\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.816122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.816063    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-lib-modules\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.816122 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.816103    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-proc\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.816250 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.816135    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-sys\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.816293 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.816245    4453 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-podres\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.916776 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916737    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-proc\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.916963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916790    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-sys\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.916963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916810    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-podres\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.916963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916855    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkqh\" (UniqueName: \"kubernetes.io/projected/2ca3b561-90f1-4c32-8ed9-40bb1e219918-kube-api-access-4vkqh\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.916963 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916892    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-sys\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.917199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916966    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-podres\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.917199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916985    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-proc\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.917199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.916971    4453 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-lib-modules\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.917199 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.917032    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ca3b561-90f1-4c32-8ed9-40bb1e219918-lib-modules\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:40.938198 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:40.938171    4453 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkqh\" (UniqueName: \"kubernetes.io/projected/2ca3b561-90f1-4c32-8ed9-40bb1e219918-kube-api-access-4vkqh\") pod \"perf-node-gather-daemonset-7cmsq\" (UID: \"2ca3b561-90f1-4c32-8ed9-40bb1e219918\") " pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:41.039510 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:41.039424    4453 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:41.349484 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:41.348466    4453 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"]
Oct 30 12:47:41.988320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:41.988286    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:42.151237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.151196    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq" event={"ID":"2ca3b561-90f1-4c32-8ed9-40bb1e219918","Type":"ContainerStarted","Data":"3e51909834d535a30416f334d9832c2b64de7f5b9928ef5aadfe792feed466a1"}
Oct 30 12:47:42.151456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.151255    4453 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq" event={"ID":"2ca3b561-90f1-4c32-8ed9-40bb1e219918","Type":"ContainerStarted","Data":"79db5d8eb43ce2721dcdaacd584bc4e87282dbfb2f9112b25811505e848f9077"}
Oct 30 12:47:42.151456 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.151277    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:42.186592 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.186537    4453 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq" podStartSLOduration=2.18652374 podStartE2EDuration="2.18652374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:47:42.185758134 +0000 UTC m=+4661.406043044" watchObservedRunningTime="2026-10-30 12:47:42.18652374 +0000 UTC m=+4661.406808650"
Oct 30 12:47:42.237974 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.237946    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8sqn_53c0c70e-e0f3-487b-9699-49f3cc352530/node-exporter/0.log"
Oct 30 12:47:42.363112 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.363088    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8sqn_53c0c70e-e0f3-487b-9699-49f3cc352530/kube-rbac-proxy/0.log"
Oct 30 12:47:42.478033 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.478001    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:42.485634 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.485614    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8sqn_53c0c70e-e0f3-487b-9699-49f3cc352530/init-textfile/0.log"
Oct 30 12:47:42.986784 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:42.986745    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tg95x_ca9eb8a5-7274-4024-841d-c26780902023/kube-rbac-proxy-main/0.log"
Oct 30 12:47:43.111554 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:43.111504    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tg95x_ca9eb8a5-7274-4024-841d-c26780902023/kube-rbac-proxy-self/0.log"
Oct 30 12:47:43.238623 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:43.238548    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-tg95x_ca9eb8a5-7274-4024-841d-c26780902023/openshift-state-metrics/0.log"
Oct 30 12:47:43.613519 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:43.613482    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:44.246388 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:44.246316    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bkwfj_d1f6c3ab-4b41-4c69-a832-d98dd617ce26/prometheus-operator/0.log"
Oct 30 12:47:44.369955 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:44.369922    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bkwfj_d1f6c3ab-4b41-4c69-a832-d98dd617ce26/kube-rbac-proxy/0.log"
Oct 30 12:47:44.507142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:44.507062    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-vhbth_232e3b46-20e1-450a-9d33-13d6c271ea30/prometheus-operator-admission-webhook/0.log"
Oct 30 12:47:45.371282 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:45.371236    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:46.501755 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:46.501717    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:47.188447 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:47.188411    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:48.169220 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:48.169191    4453 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c97ck/perf-node-gather-daemonset-7cmsq"
Oct 30 12:47:48.352587 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:48.352556    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6558dd845f-gqggd_5bea9e70-e94d-43a7-8f3f-8be36598f8fc/manager/0.log"
Oct 30 12:47:50.197237 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:50.197207    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-kpjxg_5e534504-8704-4af2-8d4c-a2bd32767adb/download-server/0.log"
Oct 30 12:47:50.204733 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:50.204705    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:51.301320 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:51.301289    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-sthrh_0f674187-2f7c-4153-9d68-b16a094df612/manager/0.log"
Oct 30 12:47:54.667425 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:54.667391    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d5gjj_38b69f25-5947-4982-8a04-bc09d51a87b3/dns/0.log"
Oct 30 12:47:54.790184 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:54.790149    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d5gjj_38b69f25-5947-4982-8a04-bc09d51a87b3/kube-rbac-proxy/0.log"
Oct 30 12:47:55.417917 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:55.417888    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h6llc_ef3a0a4d-2922-4948-af17-cf737a3f6273/dns-node-resolver/0.log"
Oct 30 12:47:56.643478 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:56.643441    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58b589f7fb-84kq8_9c34812b-7d7c-4743-bcea-78110c211de4/registry/0.log"
Oct 30 12:47:56.753651 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:56.753601    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8bfdd_cb76f4db-7292-45d7-8780-f5e14eedbfd3/node-ca/0.log"
Oct 30 12:47:56.851142 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:56.851112    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7htsq_32e04985-dc6f-43e4-ba67-2ae07d896050/discovery/0.log"
Oct 30 12:47:56.886790 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:56.886753    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-p6rqz_f404404b-d627-461a-b827-1e4e5bafa0fc/kube-auth-proxy/0.log"
Oct 30 12:47:57.018084 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:57.017998    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hpp9t_c5ef9f10-6825-4798-9579-2d1f3fed2dba/istio-proxy/0.log"
Oct 30 12:47:57.202515 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:57.202482    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-z7kbc_ec9cb544-80ec-428b-b007-bff354808519/istio-proxy/0.log"
Oct 30 12:47:57.321659 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:57.321576    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kfvvk_2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674/payload-pre-processing/0.log"
Oct 30 12:47:57.384355 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:57.384299    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 12:47:58.197187 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.197147    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7htsq_32e04985-dc6f-43e4-ba67-2ae07d896050/discovery/0.log"
Oct 30 12:47:58.269627 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.269594    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-p6rqz_f404404b-d627-461a-b827-1e4e5bafa0fc/kube-auth-proxy/0.log"
Oct 30 12:47:58.422470 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.422436    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hpp9t_c5ef9f10-6825-4798-9579-2d1f3fed2dba/istio-proxy/0.log"
Oct 30 12:47:58.637846 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.637812    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-z7kbc_ec9cb544-80ec-428b-b007-bff354808519/istio-proxy/0.log"
Oct 30 12:47:58.771100 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.771068    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kfvvk_2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674/payload-pre-processing/0.log"
Oct 30 12:47:58.838169 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:58.838138    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"
Oct 30 12:47:59.418606 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.418574    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7htsq_32e04985-dc6f-43e4-ba67-2ae07d896050/discovery/0.log"
Oct 30 12:47:59.451575 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.451495    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-775b454769-p6rqz_f404404b-d627-461a-b827-1e4e5bafa0fc/kube-auth-proxy/0.log"
Oct 30 12:47:59.584544 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.584516    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-hpp9t_c5ef9f10-6825-4798-9579-2d1f3fed2dba/istio-proxy/0.log"
Oct 30 12:47:59.773731 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.773644    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-z7kbc_ec9cb544-80ec-428b-b007-bff354808519/istio-proxy/0.log"
Oct 30 12:47:59.893467 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.893434    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-kfvvk_2d18c6a5-9f7b-4051-9ffb-cdfb6f93a674/payload-pre-processing/0.log"
Oct 30 12:47:59.970166 ip-10-0-143-18 kubenswrapper[4453]: I1030 12:47:59.970129    4453 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bqrn5_30d55454-08b7-4aca-ad04-f26a881da3d2/payload-processing/0.log"